Purpose:

To test the amount of delay (in ms) of major lines of arcade sticks as well as widely available dual mod PCBs, in comparison to a low lag control stick, when used on the Playstation 4, PlayStation 3, and XBox 360 consoles.

Equipment:

• Sony Playstation 3 console
• Sony Playstation 4 console
  XBox 360 console
  Sanwa OBSF-30 button
  1 Quick Disconnect, modified to split into two sets of raw wires allowing for simultaneous attachment to two Quick Disconnects
  Hori Real Arcade Pro VX-SA - Control stick for XBox 360 testing
  Qanba Q4RAF Black (2011 model) - Control stick for PS3 testing
• Akishop Customs PS360+ w/firmware 1.65, old build - Control for PS4 testing
  Other tested arcade sticks (various)

Basic Concept:

To test for input delay among different arcade sticks, I attached wires from one Sanwa button to two arcade stick PCBs to allow for simultaneous inputs to both sticks. I then assigned this button to Heavy attack on both the P1 and P2 sides. When the button was pressed, both characters would perform the same attack (in this case, a throw) as soon as the arcade stick PCBs were done processing the input that was sent from the single button. While the inputs would usually happen on the same frame, there are some sticks which take longer to process the inputs and it was common for these sticks to 'lose' the simulatenous throw attempt. This testing attempts to quantify the result of the frequency of losses by each brand of major arcade stick.

Above: A basic layout of the dual wiring is shown - note that two sets of QD (Quick Disconnect) wires were attached to the same button, with one set of QDs attached to each arcade stick

Preface:

In order to determine the relative amount of delay, we must first understand a few things about how sticks, consoles and fighting games handle inputs:

- Each arcade stick has inside a printed circuit board, or PCB, which handles inputs.

- When an input is made (eg. a button is pressed), an electrical signal is transmitted to the PCB, which then decodes the signal and passes it on to the game console for processing.

- Each PCB has varying amounts of time from when it receives a physical input, to when it actually sends out the input to the game console. The game console then translates the input received into an action on the screen.

- The amount of time for this whole process to occur will be referred to as 'input lag' and is what I am attempting to measure, in comparison to a control stick.

- All modern fighting games run at 60 frames per second, so any input lag will only be shown in increments of 1 frame, or 16.67 ms, for any single given trial. Therefore the only results we can expect to receive on screen are that both sticks sent inputs on the same frame (0 ms delay), or that one stick was behind by 1 frame (16.67 ms delay), behind by 2 frames (33.33 ms delay), and so forth.

- However this does not mean that the arcade stick PCBs that are being tested will have exact lag numbers of 0F, 1F, 2F, and so forth. Rather, this is the amount of lag that will be output by the game console, based on when the inputs fall within the frame polling window.

Polling Window:

As mentioned above, fighting games run at 60 FPS. Each frame has a window of 16.67 ms during which it will take inputs. However, the game console doesn't care whether it receives the input at the very start (0 ms) or very end (16.66 ms) of the window - both will output on the game screen under the same frame. Therefore it is entirely possible (but extremely unlikely - more on this below) for any single given test, for there to be nearly an entire frame of delay between two inputs yet both can display on the same frame in game.

Above: A visual representation of the input polling window over one frame (16.67 ms)

However, it is important to note that for all intents and purposes, there is no way to determine where in the input window your input will fall. This is because there is no way for us to determine when the first frame ends and the next begins, as there are no tools or indicators that show at what point during the frame an input was pressed. Because of this, we can assume that each input that is tested falls within a 'random' position in the input polling frame.

Additionally, it is exactly this 'randomness' that allows us to see variances in stick input timings, since it is much more likely for there to be variances among sticks when both inputs (simultaneously pressed) fall towards the end of the polling window.

Above: A visual representation of what happens when two inputs, one with 6 ms delay, fall near the beginning of the input polling window

Above: A visual representation of what happens when two inputs, one with 6 ms delay, fall near the end of the input polling window

Methodology:

From the above you should see that we can only determine lag in 1 frame increments. However, what we can also determine is the frequency by which one stick lags over another. Since the placement of the inputs in the above 1 frame window is essentially random, we can take a large sample of inputs and average the results in order to determine approximately how much lag each stick PCB has.

In order to grind down to a meaningful number, at least a hundred trials are needed, and to be even more precise, several hundred. In most of my tests, I ran 1,000 trials, although some PCBs were limited to 500 due to time restraints.

And, in order for the results to be comparable, we need to measure delay against a standard, or a 'control' stick. In my case, I quickly found from research and initial results that the controls to be used for testing were as follows:

PS4 control: PS360+ original PCB - PCB which was found to have the lowest lag on PS4 during the majority of my testing, eventually beaten out by the new PS360+ v.2.10 PCB released in late 2014.

PS3 control: Qanba Q4RAF Black (original) - stick which was found to have the lowest lag on PS3 during the majority of my testing, eventually beaten out by the new PS360+ v.2.10 PCB released in late 2014.

360 control: Hori Real Arcade Pro VX-SA - lowest lag arcade stick on XBox 360 which has been proven previously to perform exceptionally well in previous testing.
See: http://ffr41.air-nifty.com/faf_saf_v/2010/10/real-arcade-pro.html

Now, in order to have a reliable test, several conditions needed to be met. These conditions were:

Condition 1. Both inputs need to arrive simultaneously to two PCBs.
This was addressed by wiring one button to both PCBs of the sticks that were tested. There is effectively no additional lag from the electrical signals, and even if there were, both lengths of wire are the same distance so there should be no net change in delay.

Condition 2. The console needs to poll for inputs from both USB ports at exactly the same time.
This condition is impossible to meet as there is no way to ensure this is occurring. However, this would still happen even if you were performing this test by hand, or if two players pressed a button at the same time in a match - so there is no real way to control this. The variance from input polling on the console USB side is expected to be random and not favour either P1 or P2 side in the long run. Even so, most tests (all of those with 1,000 trials) were performed half on the P1 side and half on the P2 side with no discernible difference in results.

Condition 3. It needs to be absolutely clear whether both inputs arrive at the same time, or are 1 frame delayed, or 2 frames delayed.
This condition was met by using the Throw Break system in Guilty Gear XX Accent Core +R. In this game, when both players throw on the same frame, a very clear separation of both characters occurs. If one player has an input that is 1 frame delayed, a very clearly different animation plays - it looks like the slower player is being thrown, but then they tech out a short moment afterwards. In the case of the slower player having a 2F delay in their input, the throw is simply not teched and damage is dealt.

Above: Both players' inputs arrive on the same frame - neutral throw break occurs

 

Above: Player 2's inputs are one frame later than Player 1 - delayed throw break occurs

 

Above: Player 2's inputs are two frames later than Player 1's - no throw break occurs, Player 2 takes damage

 

See this video for an example of all three situations of 0F, 1F and 2F delay, as shown via the GGXXAC+R throw tech system.

What about 3 frames of delay? It's true that the above test is only good for showing 0F, 1F, and 2F delay frames. To make sure that there were no instances of 3F of delay, sticks with particularly high latency (Mad Catz Fight Stick Pro on PS3 and PS360+ on 360) were re-measured using a 3F delay test in GGXXAC+R and none were shown to ever show a delay of 3 frames over several hundred trials. At this point I stopped testing and concluded that the probability of such a result to be near zero or small enough to be negligible.

For Playstation 4 testing, the game used was Guilty Gear Xrd. Both characters were again, set to Sol, with the same normal attack (standing Kick) being mapped to the single button attached across both sticks. The results were then tallied based on whether the hits traded, or one side got hit first. A 2-frame delay test was also performed whereby the slower stick used standing Kick and the faster stick used standing Punch, with the latter attack being slower by 1 frame - so if there was 2F of delay, the faster stick would still win out. 2-frame delay is quite rare on PS4, only occurring in the newest line of Hori sticks (HRAP V & VLX Kuro) as well as on the Dual Shock 4 when under heavy interference from 2 other Dual Shock 4s nearby (see Results page).

Condition 4. A large enough sample needs to be taken to make the results comparable.

This condition was met as 1,000 trials were done for most sticks, with a select few limited to 500 trials. However from looking at the raw spreadsheet data, you will see that the trials are done in sets of 50, and for many sticks from as little as even 50 trials, a rough approximation of input delay can be seen that is consistent among each further set of 50 trials. Some sticks do show a larger variance than others however, which is also something to consider when comparing the reliability of each PCB.

Reason for variances

An issue that came up during testing is that if both sticks are receiving input at exactly the same time, then there shouldn't be any situations where one stick beats out the other, only to get beat out by the same stick later on - since this would indicate a potential 2 frames of variance. However, judging by the pattern of times this occurs on each set of tests, it's clear that there is a consistent pattern in how often this occurs. For example, look at the Qanba Q4RAF Black vs TE Kitty test, for PS3. While both sticks are at the top of their class for input delay, each stick occasionally (1% of the time) wins out over the other. In these cases, my best guess is that this may be due to the inputs arriving at the edge of the 1F polling window, and the console polls one port before it polls the other, leading one of the two sticks to win out. In the long run, the console doesn't favour either side, since it could poll the P1 side 1 ms before polling the P2 side, or vice versa, depending on when the input arrives in the USB polling window.

However this does not explain why certain PCBs, like the PS360+ when used on 360, have such variable delay. It could be that the 360 polling rates are much slower, or it could be that the PS360+ has variable delay in processing its inputs on the 360 side - both are possibilities here.

Additional Support - Same PCB

There is additional evidence to support the reliability of this test. The Qanba line of sticks, apart from the original Q4RAF Black (which performs very well on both consoles) uses the same PCB. All of these sticks have nearly identical amounts of delay on both consoles, suggesting that the testing method I have used is very consistent, and accurate to within fractions of a millsecond.

The following Qanba sticks appear to have identical PCBs:

• - Qanba Q4RAF Ice Blue - 3.98 ms on 360, 13.95 ms on PS3
• - Qanba Q4RAF White/Red - 3.92 ms on 360, 14.27 ms on PS3
• - Qanba Q2 Pro White Solid - 3.97 ms on 360, 14.23 ms on PS3

Additionally, two Mad Catz FightStick Pros were tested on XBox 360:

• - Mad Catz FightStick Pro w/ChImp dual mod - 6.67 ms
• - Mad Catz FightStick Pro - 6.68 ms

Alternative Testing & Support

Roughly a year before I began testing using one button wired to two PCBs, I attempted to perform input lag testing by hand. That is, using one hand to hit a button on one stick, and my other hand to hit the same button on another, on the same frame. I figured I would do it just to get some ballpark numbers and that hopefully with enough trials I would end up with fairly consistent number for input delay on a couple of my sticks.

This certainly does seem unscientific at first, but consider that several games require you to hit two buttons simultaneously on the same frame. 3rd Strike and KoFXIII both have EX moves with no input leniency, and players are used to performing multiple button presses on the same frame without issue.

I used a similar methodology as above when doing the manual tests, with the following exceptions:

- I used KoF XIII as the test game, simply hit a close C on both sticks simultaneously by hand.

- Due to the time intensive nature I only did 300 trials per stick

- In order to avoid any sort of mental bias towards one stick or the other, I did this while looking away from the screen and recorded my results via HDPVR, which I later transcribed.

- To account for any manual dexterity differences in my hands, I switched hands halfway through (at the 150 mark).

The results of this manual testing is surprising to look back on now. As you can see above, the results come out very, very close to the automated button test, as both sticks tested (the Joytron EXChanger and Qanba Q4RAF) are within 0.5 ms of the actual results. I would argue that manual testing is indeed a reliable way to roughly determine PCB lag, given that you perform enough trials and ensure you're using both hands equally on each side. Testing blindly also helps ensure you aren't subconsciously biased towards any one stick.

The results of the manual testing are shown on the "Raw Data" page after the Results.

NEXT PAGE: Results

LAST PAGE: RAW DATA

Site History:

03/08/2016: Major Update (PS4, PS3)

- Updated results and raw data to include results for several new PS4 and PS3 sticks:

- Added data for Mad Catz TE2+ (PS4) for PS4 and PS3 in both PS3 and PS4 modes
- Added data for Mad Catz TE-S+ (PS4) for PS4 and PS3 in both PS3 and PS4 modes
- Added data for CronusMax Plus adapter for 360 to PS3, PS3 to 360, PS3 to PS4, and 360 to PS4
- Added data for Brook Fighting Board (PS4 PCB)
- Added data for Brook Super Converter (PS3 to PS4)
- Added data for Mad Catz TE MLG FightStick (PS3)

07/26/2015:

- Updated Hori Real Arcade Pro 4 series PCB results to indicate all models of sticks with the same PCB (HRAP4 Kai, HRAP V / Silent, DoA5LR)

07/11/2015: Major Update (PS4, PS3)

- Updated results and raw data to include results for several new PS4 and PS3 sticks:

- Updated data on Hori Real Arcade Pro v4 on PS3 (PS3 mode) to 1000 trials
- Added data for Hori Real Arcade Pro v4 (PS3) for PS4 mode
- Added data for Mad Catz TE2 (PS3) for both PS3 and PS4 modes
- Added data for Guilty Gear Xrd Custom Fighter's Pad (PS3) for both PS3 and PS4 modes
- Added data for Guilty Gear Xrd Custom Fighter's Pad (PS4)
- Added data for Hori Fighting Commander 4 (PS3) for both PS3 and PS4 modes
- Added data for Hori Fighting Commander 4 (PS4)
- Added data for Hori RAP4 Premium VLX Kuro (PS3) for both PS3 and PS4 modes
- Added data for Hori RAP4 Premium VLX Kuro (PS4)
- Added data for Hori RAP V Silent (PS3) for both PS3 and PS4 modes
- Added data for Hori RAP V Silent (PS4)
- Clarified difference between Qanba Q4RAF Black old and new models on Results page
- Also added average delay for the new line of Qanba Q4RAF Black & Eightarc Fusion sticks

01/25/2015: PS4 sections added

- Updated overview page to include PS4 testing methodology
- Updated results and raw data to include results for several new PS4, PS3, and 360 sticks:

- Added data for Akishop PS360+ 1.65 old build (PS4)
- Added data for Sony Dual Shock 4 (PS4)
- Added data for Sony Dual Shock 4 (PS3 mode)
- Added data for Hori Real Arcade Pro v4 (PS4)
- Added data for Hori Real Arcade Pro v4 (PS3 mode)
- Added data for Mad Catz TE 2 (PS4)
- Added data for PS3 -> 360 Cronus adapter with low lag (+0.13 ms) PS360+ PS3 mode stick (XBox 360)
- Added data for PS3 -> 360 Cronus adapter with high lag (+21.38 ms) Mad Catz Fightstick Pro PS3 stick (XBox 360)
- Added data for 360 -> PS3 Cronus adapter with control stick (+0 ms) HRAP VX-SA 360 stick (PS3)

07/25/2014:

- Updated overview page to improve wording of Preface and other sections
- Updated results and raw data to include the following new sticks:

- Added data for Phreakmods Cerberus (PS3)
- Added data for Sega Virtua Stick High Grade (PS3)
- Added data for Qanba Q1 (PS3)
- Added data for Mad Catz TE Round 1(PS3)
- Added data for Akishop Customs PS360+ New build (XBox 360)
- Added data for Mad Catz TE Round 2 (XBox 360)
- Added data for Mad Catz TE-S Chun Li (XBox 360)
- Added data for Mad Catz FightStick V.S. (XBox 360)

07/16/2014:

- Changed PS3 rankings to set the PS3 control stick, the Qanba Q4 Black (original PCB), to 0.00 ms, re-ranked all PS3 sticks accordingly
- Added a 95% confidence interval calculation which provides us with a range in which 95% of the data sets will fall within, relative to the Delay (ms) reported. This is not a measure of input lag variance from stick to stick, but rather a number that represents how accurate the average reported delay (ms) is in 95% of the recorded trials.
- Added calculations for mean and standard deviation (shown in Raw Data only)

- Added data for PS360+ firmware version 1.4 (XBox 360)
- Added data for Mad Catz FightStick Pro (XBox 360)
- Corrected rating for Toodles Cthulhu (PS3); incorrectly had the 'Tie' and 'Wins' columns for the Cthulu transposed

- Added Site History & Contact details sections

07/07/2014:

- Site posted - Overview, Results and Raw Data uploaded

Contact:

@TeyahDL, or at http://www.dustloop.com/forums or http://forums.shoryuken.com/