Twitter header: part 3
Intro
This is the final part of a 3 part series covering the X-Client-Transaction-Id header on Twitter. Part 3 is going to be more technical than the olther 2 parts, as we will be making the header generator in golang. This will require a lot more work than the other 2 parts.
Animation
In part 2, we learned what all goes into building this header. During that adventure, we saw the biggest thing they could validate is the sha256 hash and the only part of that hash that is dynamic is the animation related stuff. Therefore, we know we will have to replicate this:
let newDiv = document["createElement"]("div")
document["body"]["append"](newDiv)
let r = newDiv["animate"]({
color: ["#43e096", "#646538"],
transform: ["rotate(0deg)", "rotate(342deg)"],
easing: "cubic-bezier(0.33,0.10,0.29,-0.01)"
}, 4096);
r["pause"]()
r["currentTime"] = 2000;
let style = getComputedStyle(newDiv)
The expected output would be style.color = "rgb(77, 187, 122)" and style.transform = "matrix(-0.227212, 0.973845, -0.973845, -0.227212, 0, 0)".
Figuring out how to replicate these outputs in GoLang will require us to understand exactly what this animate function does. I decided I was going to look at whatever commit added this feature into chromium, aka the chrome browser itself. I was hoping this would make it easier to find what code I need to look at in the source code, since it's quite a big repository.
After some looking I found this post on developer.chrome.com showing exactly when this feature was added. If you scroll down to the bottom, you see mentions of this web-animations-js GitHub repository that they say the functionality is based off of. This is a great lead, however, it doesn't really solve our problem. Firstly, this post is almost a decade old, who knows if this post is still accurate to how the function works now. Secondly, this repository is as well very old, very outdated, and pretty hard for me to read, it's not using very "modern" code styles to say the very least.
First let's start by trying to confirm if this "web-animations-js" code is still reflective of how the modern day Element.animate function works. To do this, I'm going to clone the repo and setup a test html file to import all the JS files it uses and execute the code segment I gave earlier to compare the outputs. This seems easy, however, they don't provide any good testing HTML for this project and I'm not a web developer so I'm not very sure how to properly setup a test environment for this. After some trial and error, I was able to make a proper test file for this though:
<head>
<script src="src/dev.js"></script>
<script src="src/scope.js"></script>
<script src="src/handler-utils.js"></script>
<script src="src/timing-utilities.js"></script>
<script src="src/interpolation.js"></script>
<script src="src/property-interpolation.js"></script>
<script src="src/normalize-keyframes.js"></script>
<script src="src/keyframe-interpolations.js"></script>
<script src="src/number-handler.js"></script>
<script src="src/box-handler.js"></script>
<script src="src/dimension-handler.js"></script>
<script src="src/font-weight-handler.js"></script>
<script src="src/position-handler.js"></script>
<script src="src/shadow-handler.js"></script>
<script src="src/transform-handler.js"></script>
<script src="src/visibility-handler.js"></script>
<script src="src/animation.js"></script>
<script src="src/timeline.js"></script>
<script src="src/apply-preserving-inline-style.js"></script>
<script src="src/apply.js"></script>
<script src="src/web-animations-next-animation.js"></script>
<script src="src/color-handler.js"></script>
<script src="src/deprecation.js"></script>
<script src="src/element-animatable.js"></script>
<script src="src/effect-callback.js"></script>
<script src="src/group-constructors.js"></script>
<script src="src/keyframe-effect-constructor.js"></script>
<script src="src/keyframe-effect.js"></script>
<script src="src/matrix-decomposition.js"></script>
<script src="src/matrix-interpolation.js"></script>
<script src="src/property-names.js"></script>
<script src="src/tick.js"></script>
<script src="src/web-animations-bonus-cancel-events.js"></script>
<script src="src/web-animations-bonus-object-form-keyframes.js"></script>
</head>
After making this file, I used the VSCode plugin live server to easily spin-up this html file into a local-host "website", for testing. I do want to make sure that their code is correctly overriding the built-in animate function:
We can see that the animate function is indeed being overwritten with the web-animations-js code, so now we can test if the results are 1:1. From the output, we see that they are!
Now we can start figuring this out. Like I said previously, this code is hard for me to read, I opted to set breakpoints in a few parts of the code until I could figure out what exactly I needed. Firstly, I needed to identify how r["currentTime"] = 2000; was changing the color and transform. After some looking, I found web-animations-next-animation.js which contains a "getter" and "setter" for currentTime. I can set a breakpoint on the "setter" and see what when I do set the currentTime, this code is indeed hit:
set currentTime(v) {
this._updatePromises();
this._animation.currentTime = isFinite(v) ? v : Math.sign(v) * Number.MAX_VALUE;
this._register();
this._forEachChild(function(child, offset) {
child.currentTime = v - offset;
});
this._updatePromises();
}
I went ahead and stepped through this code, line by line, waiting to see at what point getComputedStyle(newDiv).color would change. It only changed after this._animation.currentTime = isFinite(v) ? v : Math.sign(v) * Number.MAX_VALUE;. The same logic should apply, whatever this._animate._currentTime is must have a setter that's updating color/transform. Using chrome's very good devtools, we can trace this very nicely. Let's see the _animate._currentTime setter:
set currentTime(newTime) {
newTime = +newTime;
if (isNaN(newTime))
return;
scope.restart();
if (!this._paused && this._startTime != null) {
this._startTime = this._timeline.currentTime - newTime / this._playbackRate;
}
this._currentTimePending = false;
if (this._currentTime == newTime)
return;
if (this._idle) {
this._idle = false;
this._paused = true;
}
this._tickCurrentTime(newTime, true);
scope.applyDirtiedAnimation(this);
}
After some looking, we see that scope.applyDirtiedAnimation(this) is what's responsible for modifying the Element's attributes, however, that function just calls a function that isn't exactly clear what it's calling, if you look into it though, you can see that this is the function it's running:
scope.KeyframeEffect = function(target, effectInput, timingInput, id) {
var effectTime = EffectTime(shared.normalizeTimingInput(timingInput));
var interpolations = scope.convertEffectInput(effectInput);
var timeFraction;
var keyframeEffect = function() {
WEB_ANIMATIONS_TESTING && console.assert(typeof timeFraction !== 'undefined');
interpolations(target, timeFraction);
};
keyframeEffect._update = function(localTime) {
timeFraction = effectTime(localTime);
return timeFraction !== null;
};
keyframeEffect._clear = function() {
interpolations(target, null);
};
keyframeEffect._hasSameTarget = function(otherTarget) {
return target === otherTarget;
};
keyframeEffect._target = target;
keyframeEffect._totalDuration = effectTime._totalDuration;
keyframeEffect._id = id;
return keyframeEffect;
};
We see what we really want is this keyframeEffect function. This function calls interpolations(target, timeFraction);. If you trace that call, you get here:
scope.convertEffectInput = function(effectInput) {
var keyframes = shared.normalizeKeyframes(effectInput);
var propertySpecificKeyframeGroups = makePropertySpecificKeyframeGroups(keyframes);
var interpolations = makeInterpolations(propertySpecificKeyframeGroups);
return function(target, fraction) {
if (fraction != null) {
interpolations.filter(function(interpolation) {
return fraction >= interpolation.applyFrom && fraction < interpolation.applyTo;
}).forEach(function(interpolation) {
var offsetFraction = fraction - interpolation.startOffset;
var localDuration = interpolation.endOffset - interpolation.startOffset;
var scaledLocalTime = localDuration == 0 ? 0 : interpolation.easingFunction(offsetFraction / localDuration);
scope.apply(target, interpolation.property, interpolation.interpolation(scaledLocalTime));
});
} else {
for (var property in propertySpecificKeyframeGroups)
if (property != 'offset' && property != 'easing' && property != 'composite')
scope.clear(target, property);
}
};
};
Specifically, we're going to end up on the scope.apply(target, interpolation.property, interpolation.interpolation(scaledLocalTime)); which actually sets the Element's color/transformation. We can see that because interpolation.property is color when we look. So color is set to interpolation.interpolation(scaledLocalTime) and scaledLocalTime is the return of interpolation.easingFunction(offsetFraction / localDuration). It appears this input looks like a percentage of how far into the total animation lifespan we're setting the currentTime to. So if the total time is 4096 and our currentTime is 2000, the input to this should be 2000 / 4096 aka 0.48828125. So first we need to look into this easingFunction, which is you remember, is actually one of our inputs to the animate function, easing: "cubic-bezier(0.33,0.10,0.29,-0.01)". Let's look into the easing function:
function cubic(a, b, c, d) {
if (a < 0 || a > 1 || c < 0 || c > 1) {
return linear;
}
return function(x) {
if (x <= 0) {
var start_gradient = 0;
if (a > 0)
start_gradient = b / a;
else if (!b && c > 0)
start_gradient = d / c;
return start_gradient * x;
}
if (x >= 1) {
var end_gradient = 0;
if (c < 1)
end_gradient = (d - 1) / (c - 1);
else if (c == 1 && a < 1)
end_gradient = (b - 1) / (a - 1);
return 1 + end_gradient * (x - 1);
}
var start = 0, end = 1;
function f(a, b, m) { return 3 * a * (1 - m) * (1 - m) * m + 3 * b * (1 - m) * m * m + m * m * m};
while (start < end) {
var mid = (start + end) / 2;
var xEst = f(a, c, mid);
if (Math.abs(x - xEst) < 0.00001) {
return f(b, d, mid);
}
if (xEst < x) {
start = mid;
} else {
end = mid;
}
}
return f(b, d, mid);
}
}
This function cubic is taking in 0.33, 0.1, 0.29, -0.01, the same numbers we see in the input object easing: "cubic-bezier(0.33,0.10,0.29,-0.01)". The return function is what inputs this time percentage and outputs whatever number we need. Great, now let's look into how this output is being transformed into what we see, the RGBA value's. We end up here:
var interp = scope.Interpolation.apply(null, interpolationArgs);
return function(t) {
if (t == 0) return left;
if (t == 1) return right;
return interp(t);
};
Eventually, after a lot of looking, we end up here:
function interpolate(from, to, f) {
if ((typeof from == 'number') && (typeof to == 'number')) {
return from * (1 - f) + to * f;
}
if ((typeof from == 'boolean') && (typeof to == 'boolean')) {
return f < 0.5 ? from : to;
}
WEB_ANIMATIONS_TESTING && console.assert(
Array.isArray(from) && Array.isArray(to),
'If interpolation arguments are not numbers or bools they must be arrays');
if (from.length == to.length) {
var r = [];
for (var i = 0; i < from.length; i++) {
r.push(interpolate(from[i], to[i], f));
}
return r;
}
throw 'Mismatched interpolation arguments ' + from + ':' + to;
}
When looking, from and to are both arrays of numbers and f is 0.3015547171021191, aka what our curve function is outputting. When looking closer, from and to appear to align with the numbers in the let numArr = [67,224,150,100,101,56,240,84,140,73,126] before they're turned to hex and used for the animation function as a hex color code.
So we have what we need, if you debug that interpolate function, you can see that it does the same thing for rotation aka transform. Now we can start making the generator!
Porting Code
When porting this code, we will need to basically make everything a float64. Porting this code does appear easy, so let's go ahead and see!
Firstly, we'll port the javascript we just got to generate the RGBA and matrix numbers. First we need to port that cubic easing code, this looks like a pretty simple algorithm, porting this code is overall pretty simple:
// file: ./payload/cubiccurve.go
package payload
type cubic struct {
Curves [4]float64
}
func (c *cubic) getValue(time float64) float64 {
startGradient := 0.0
endGradient := 0.0
if time <= 0.0 {
if c.Curves[0] > 0.0 {
startGradient = c.Curves[1] / c.Curves[0]
} else if c.Curves[1] == 0.0 && c.Curves[2] > 0.0 {
startGradient = c.Curves[3] / c.Curves[2]
}
return startGradient * time
}
if time >= 1.0 {
if c.Curves[2] < 1.0 {
endGradient = (c.Curves[3] - 1.0) / (c.Curves[2] - 1.0)
} else if c.Curves[2] == 1.0 && c.Curves[0] < 1.0 {
endGradient = (c.Curves[1] - 1.0) / (c.Curves[0] - 1.0)
}
return 1.0 + endGradient*(time-1.0)
}
start := 0.0
end := 1.0
mid := 0.0
for start < end {
mid = (start + end) / 2
xEst := f(c.Curves[0], c.Curves[2], mid)
if abs(time-xEst) < 0.00001 {
return f(c.Curves[1], c.Curves[3], mid)
}
if xEst < time {
start = mid
} else {
end = mid
}
}
return f(c.Curves[1], c.Curves[3], mid)
}
func abs(in float64) float64 {
if in < 0 {
return -in
}
return in
}
func f(a, b, m float64) float64 {
return 3.0*a*(1-m)*(1-m)*m + 3.0*b*(1-m)*m*m + m*m*m
}
You can see that everything here is a float64, we will be doing that throughout everywhere we can. Next we need to port the interpolate function:
// file: ./payload/interpolate.go
package payload
func interpolate(from, to []float64, f float64) []float64 {
out := []float64{}
for i := 0; i < len(from); i++ {
out = append(out, interpolateNum(from[i], to[i], f))
}
return out
}
func interpolateNum(from, to, f float64) float64 {
return from*(1-f) + to*f
}
Since using the any or interface{} type in golang is pretty annoying, I went ahead and split it into 2 funcs so we don't need to check for boolean or number type.
Now the last thing, the value that is returned for matrix is actually the degrees of rotation, not the matrix, which is 6 numbers that represent a 2d plain of how the image is rotated. In order to convert degrees to matrix, I looked on stackoverflow and found this answer: stackoverflow.com/a/25367782. You may notice that this uses radians though, I'm not sure what a radian is so I had to google converting from degrees to radians, turns out that it's actually super easy, google itself will give you the formula which is 1° × π/180. Now we know everything we need to write that last part:
// file: ./payload/rotation.go
package payload
import "math"
func convertRotationToMatrix(degrees float64) []float64 {
// ! first convert degrees to radians
radians := degrees * math.Pi / 180
// ! now we do this:
/*
[cos(r), -sin(r), 0]
[sin(r), cos(r), 0]
in this order:
[cos(r), sin(r), -sin(r), cos(r), 0, 0]
*/
c := math.Cos(radians)
s := math.Sin(radians)
return []float64{c, s, -s, c, 0, 0}
}
This is only one part of the payload though, there's still everything else. Now is when we make the actual header generator.
So let's port code sequentially, let's assume our inputs will be path, method, key all as strings, then frames as a type [][][]int. This will represent all the 4 x-animation-0 arrays we need.
First step would be converting the key to bytes. Let's see how their js does it:
function makeUintArray(key) {
return new Uint8Array(atob(key)["split"]("")["map"](n => n["charCodeAt"](0)))
}
So it passes that string into this function that base64 decodes it but maps all of the decoded characters to their ascii character code. Golang doesn't have an stdlib function for this, not as far as I know, so let's make our own function for charCodeAt:
// file: ./payload/utils.go
func atob(input string) string {
data, err := base64.RawStdEncoding.DecodeString(input)
if err != nil {
return ""
}
return string(data)
}
func charCodeAt(a string, i int) int {
return int(a[i])
}
To make this all easier to read, I've made a helper func called atob as well, just to match what we see in javascript. So now this is the beginning of our generator:
// file: ./payload/payload.go
// GenerateHeader - generates an x-transaction-id header
func GenerateHeader(path, method, key string, frames [][][]int) string {
keyBytes := []int{}
key = atob(key)
for i := 0; i < len(key); i++ {
keyBytes = append(keyBytes, charCodeAt(key, i))
}
return ""
}
Now we need to do time. The issue is, they represent time in a 4 byte representation of uint8's instead of just one single number. I most definitely don't know how to do that in golang already, time to google! Eventually, I came across this fairly old gist on GitHub that did exactly that: gist.github.com/chiro-hiro/.... We now have what we need to do the time bytes:
// file: ./payload/utils.go
func timeToBytes(val uint32) []int {
r := make([]int, 4)
for i := uint32(0); i < 4; i++ {
r[i] = int((val >> (8 * i)) & 0xff)
}
return r
}
// file: ./payload/payload.go
timeNow := uint32((time.Now().UnixMilli() - 1682924400*1000) / 1000)
timeNowBytes := timeToBytes(timeNow)
Now the next thing the header gen does is generate that X_LOGO_HEX_STR. This part should be made relatively easy now that we have matrix and RGBA generation done, there is a few issues though.
The first problem we end up facing is something I really didn't expect. So what this is doing is basically taking the RGB numbers and the matrix numbers then converting them to hex, then removing any . or - left over. Seems simple, except there's a difference between how GoLang and JavaScript do hex with floating point numbers. I was never really able to figure out why they do things differently, but we do know what we need to do overall is convert a float64 to a hex string. I looked on GitHub for something in GoLang that could convert a float64 to hex string, eventually I found this: github.com/jeffreydwalter/arlo-go/...
func FloatToHex(x float64) string {
var result []byte
quotient := int(x)
fraction := x - float64(quotient)
for quotient > 0 {
quotient = int(x / 16)
remainder := int(x - (float64(quotient) * 16))
if remainder > 9 {
result = append([]byte{byte(remainder + 55)}, result...)
} else {
for _, c := range strconv.Itoa(int(remainder)) {
result = append([]byte{byte(c)}, result...)
}
}
x = float64(quotient)
}
if fraction == 0 {
return string(result)
}
result = append(result, '.')
for fraction > 0 {
fraction = fraction * 16
integer := int(fraction)
fraction = fraction - float64(integer)
if integer > 9 {
result = append(result, byte(integer+55))
} else {
for _, c := range strconv.Itoa(int(integer)) {
result = append(result, byte(c))
}
}
}
return string(result)
}
For anyone concerned about using code off of GitHub, the license used on this project is the MIT license, which makes it free for anyone to use for pretty much anything they want, including private use. I've shortened this a lot, however, I probably spent over an hour figuring out both what was wrong and what I needed to do to even be able to get the idea of finding this repository. This was quite a headache, I wasn't aware of how exactly converting from string to hex actually worked, much less float64 to hex.
Anyways, now that we've done that, we can finish up making that X_LOGO_HEX_STR code:
// ./file/payload/util.go
func floatToHex(x float64) string {
var result []byte
//...
return string(result)
}
func round(num float64) int {
return int(num + math.Copysign(0.5, num))
}
func toFixed(num float64, precision int) float64 {
output := math.Pow(10, float64(precision))
return float64(round(num*output)) / output
}
func a(b, c, d float64) float64 {
return b*(d-c)/255 + c
}
func b(a int) float64 {
if a%2 == 1 {
return -1.0
}
return 0.0
}
// ./file/payload/payload.go
row := frames[keyBytes[5]%4][keyBytes[2]%16]
targetTime := float64(keyBytes[12]%16*(keyBytes[14]%16)*(keyBytes[7]%16)) / totalTime
fromColor := []float64{float64(row[0]), float64(row[1]), float64(row[2]), 1.0}
toColor := []float64{float64(row[3]), float64(row[4]), float64(row[5]), 1.0}
fromRotation := []float64{0.0}
toRotation := []float64{a(float64(row[6]), 60.0, 360.0)}
row = row[7:]
curves := [4]float64{}
for i := 0; i < len(row); i++ {
curves[i] = a(float64(row[i]), b(i), 1.0)
}
c := &cubic{Curves: curves}
val := c.getValue(targetTime)
color := interpolate(fromColor, toColor, val)
rotation := interpolate(fromRotation, toRotation, val)
matrix := convertRotationToMatrix(rotation[0])
strArr := []string{}
for i := 0; i < len(color)-1; i++ {
strArr = append(strArr, hex.EncodeToString([]byte{byte(math.Round(color[i]))}))
}
for i := 0; i < len(matrix)-2; i++ {
rounded := toFixed(matrix[i], 2)
if rounded < 0 {
rounded = -rounded
}
strArr = append(strArr, "0"+strings.ToLower(floatToHex(rounded)[1:]))
}
strArr = append(strArr, "0", "0")
This is quite a lot more code than you may have expected, however, it's all just stuff we already knew. Like making the time used for the cubic-curve generation, we already knew that was x/4096. We knew x was something from the key bytes, multiplied together. I did go ahead and port those a and b functions as well though, since those are just math, I figured I didn't need to go into it much. I also wrote some helper functions to do fixed floating point precision a little easier to read. We do need to fix the floats to x.xx percision though, by default the float64's will return x.xxxxxxxxxxxxxxxx which would mess with our output in hex.
Next we will need to do the sha256 hash, this is pretty easy in golang, however, the plan I have in mind is to keep the byte numbers as int instead, it'll make it easier for me to read personally since that's how the JavaScript uses them:
// file: ./payload/payload.go
hash := sha256.Sum256([]byte(fmt.Sprintf(`%s!%s!%vbird%s`, method, path, timeNow, strings.Join(strArr, ""))))
hashBytes := []int{}
for i := 0; i < len(hash)-16; i++ {
hashBytes = append(hashBytes, int(hash[i]))
}
Now we just need to get a random number, 0-256, which will act as the xorByte. Then we will combine all of the "bytes" (aka int's in this case) into one array so we can xor them:
xorByte := rand.Intn(256)
bytes := []int{xorByte}
bytes = append(bytes, keyBytes...)
bytes = append(bytes, timeNowBytes...)
bytes = append(bytes, hashBytes...)
bytes = append(bytes, 1)
We're actually super close to being done, only two more things. Firstly, we need to XOR these int's then convert them to bytes in a byte array, this is so that when we go to get the final product, base64 encoded string, we can pass in []byte like it wants:
// file: ./payload/payload.go
out := []byte{}
for i := 0; i < len(bytes); i++ {
if i == 0 {
// ! don't xor the xor byte
out = append(out, byte(bytes[i]))
continue
}
out = append(out, byte(bytes[i]^xorByte))
}
Now we have the final step, btoa, aka base64 encoding:
// file: ./payload/utils.go
func btoa(str []byte) string {
return base64.StdEncoding.EncodeToString([]byte(str))
}
// file: ./payload/payload.go
package payload
import (
"crypto/sha256"
"encoding/hex"
"fmt"
"math"
"math/rand"
"strings"
"time"
)
const totalTime = 4096.0
// GenerateHeader - generates an x-transaction-id header
func GenerateHeader(path, method, key string, frames [][][]int) string {
// ...
xorByte := rand.Intn(256)
bytes := []int{xorByte}
bytes = append(bytes, keyBytes...)
bytes = append(bytes, timeNowBytes...)
bytes = append(bytes, hashBytes...)
bytes = append(bytes, 1)
out := []byte{}
for i := 0; i < len(bytes); i++ {
if i == 0 {
// ! don't xor the xor byte
out = append(out, byte(bytes[i]))
continue
}
out = append(out, byte(bytes[i]^xorByte))
}
return strings.ReplaceAll(btoa(out), "=", "")
}
Testing
To test our code, we're going to be using the script we made to pull information out of the header before. If we supply a path, method, key, and frames that we got from our browser, so we know it's correct, we can hard code the timeNow and xorByte that we get from that output. After hardcoding those values, the output should be 1:1 with our header in the browser.
Thankfully, it is. When I was coding this for the first time though, of-course my output wasn't correct first try, just one of the joys of programming I guess, nothing ever works first try.
Now what I wanted to do was make this into a simple localhost HTTP API to use it in a tool to check if using this header correctly would increase account quality when doing stuff on Twitter, like tweeting, liking posts, following people, etc. I'll spoil the result, it's completely useless. The header doesn't make Twitter "trust" your request any more or less. I guess it's just there to gather data right now, however, it makes me wonder; Why go through so much effort making this whole dynamic key system, obfuscating the files, etc. just to do practically nothing with it? At least nothing that would warrant obfuscation like that.
Conclusion
You can find the full project, including that simple localhost HTTP API I made on my GitHub: github.com/obfio/twitter-tid-generator!
I hope everyone enjoyed this three part series on this header, even though it didn't turn out to be a useful header. Sometimes, that's just how it goes when reverse engineering and/or botting a website. Sometimes you end up accidentally wasting time, usually it ends up being a nice learning experience though!
For anyone wondering, no I won't be posting the script I used to test the effects of the header on account quality, they aren't super hard to make though.
References
GitHub for this code: github.com/obfio/twitter-tid-generator
Gist for int32 -> int8 array: gist.github.com/chiro-hiro/...
GitHub for float64 -> hex: github.com/jeffreydwalter/arlo-go/...
GitHub for Web-Animations-JS: github.com/web-animations/...
Google Chrome post about adding web-animations: developer.chrome.com/blog/...