This guide maps most of x264's options to FFmpeg's options along with detailed descriptions by libx264 developer Dark_Shikari. FFmpeg developer superdump has implemented x264 presets to FFmpeg. You can find his guide here.
Frame-type options:
g <integer>
Keyframe interval, also known as GOP length. This determines the maximum distance between I-frames. Very high GOP lengths will result in slightly more efficient compression, but will make seeking in the video somewhat more difficult. Recommended default: 250
keyint_min <integer>
Minimum GOP length, the minimum distance between I-frames. Recommended default: 25
sc_threshold <integer>
Adjusts the sensitivity of x264's scenecut detection. Rarely needs to be adjusted. Recommended default: 40
bf <integer>
B-frames are a core element of H.264 and are more efficient in H.264 than any previous standard. Some specific targets, such as HD-DVD and Blu-Ray, have limitations on the number of consecutive B-frames. Most, however, do not; as a result, there is rarely any negative effect to setting this to the maximum (16) since x264 will, if B-adapt is used, automatically choose the best number of B-frames anyways. This parameter simply serves to limit the max number of B-frames.
Notes”
– Baseline Profile, such as that used by iPods, does not support B-frames. Recommended default: 16
– If you want to generate H.264 baseline for older iPhone 3 phones, then do not use Advanced Parameters like B-frames or else the output will be generated as Main profile.
0: Very fast, but not recommended. Does not work with pre-scenecut (scenecut must be off to force off b-adapt).
1: Fast, default mode in x264. A good balance between speed and quality.
2: A much slower but more accurate B-frame decision mode that correctly detects fades and generally gives considerably better quality. Its speed gets considerably slower at high bframes values, so its recommended to keep bframes relatively low (perhaps around 3) when using this option. It also may slow down the first pass of x264 when in threaded mode.
b_strategy <integer>
x264, by default, adaptively decides through a low-resolution lookahead the best number of B-frames to use. It is possible to disable this adaptivity; this is not recommended. Recommended default: 1
bframebias
Make x264 more likely to choose higher numbers of B-frames during the adaptive lookahead. Not generally recommended. Recommended default: 0
flags2 +bpyramid
Allows B-frames to be kept as references. The name is technically misleading, as x264 does not actually use pyramid coding; it simply adds B-references to the normal reference list. B-references get a quantizer halfway between that of a B-frame and P-frame. This setting is generally beneficial, but it increases the DPB (decoding picture buffer) size required for playback, so when encoding for hardware, disabling it may help compatibility.
coder
CABAC is the default entropy encoder used by x264. Though somewhat slower on both the decoding and encoding end, it offers 10-15% improved compression on live-action sources and considerably higher improvements on animated sources, especially at low bitrates. It is also required for the use of trellis quantization. Disabling CABAC may somewhat improve decoding performance, especially at high bitrates. CABAC is not allowed in Baseline Profile. Recommended default: -coder 1 (CABAC enabled)
refs <integer>
One of H.264's most useful features is the ability to reference frames other than the one immediately prior to the current frame. This parameter lets one specify how many references can be used, through a maximum of 16. Increasing the number of refs increases the DPB (Decoded Picture Buffer) requirement, which means hardware playback devices will often have strict limits to the number of refs they can handle. In live-action sources, more reference have limited use beyond 4-8, but in cartoon sources up to the maximum value of 16 is often useful. More reference frames require more processing power because every frame is searched by the motion search (except when an early skip decision is made). The slowdown is especially apparent with slower motion estimation methods. Recommended default: -refs 6
flags
Disable loop filter. Recommended default: -flags +loop (Enabled)
deblockalpha <integer>
deblockbeta <integer>
One of H.264's main features is the in-loop deblocker, which avoids the problem of blocking artifacts disrupting motion estimation. This requires a small amount of decoding CPU, but considerably increases quality in nearly all cases. Its strength may be raised or lowered in order to avoid more artifacts or keep more detail, respectively. Deblock has two parameters: alpha (strength) and beta (threshold). Recommended defaults:-deblockalpha 0 -deblockbeta 0 (Must have '-flags +loop')
Ratecontrol:
cqp <integer>
Constant quantizer mode. Not exactly constant completely–B-frames and I-frames have different quantizers from P-frames. Generally should not be used, since CRF gives better quality at the same bitrate. b <integer>
Enables target bitrate mode. Attempts to reach a specific bitrate. Should be used in 2-pass mode whenever possible; 1-pass bitrate mode is generally the worst ratecontrol mode x264 has.
crf <float>
Constant quality mode (also known as constant ratefactor). Bitrate corresponds approximately to that of constant quantizer, but gives better quality overall at little speed cost. The best one-pass option in x264.
maxrate <integer>
Specifies the maximum bitrate at any point in the video. Requires the VBV buffersize to be set. This option is generally used when encoding for a piece of hardware with bitrate limitations.
bufsize <integer>
Depends on the profile level of the video being encoded. Set only if you're encoding for a hardware device.
rc_init_occupancy <float>
Initial VBV buffer occupancy. Note: Don't mess with this.
qmin <integer>
Minimum quantizer. Doesn't need to be changed. Recommended default: -qmin 10
qmax <integer>
Maximum quantizer. Doesn't need to be changed. Recommended default: -qmax 51
qdiff <integer>
Set max QP step. Recommended default: -qdiff 4
bt <float>
Allowed variance of average bitrate
i_qfactor <float>
Qscale difference between I-frames and P-frames. Note: -i_qfactor is handled a little differently than –ipratio. Recommended: -i_qfactor 0.71
b_qfactor <float>
Qscale difference between P-frames and B-frames.
chromaoffset <integer>
QP difference between chroma and luma.
pass <1,2,3>
Used with –bitrate. Pass 1 writes the stats file, pass 2 reads it, and 3 both reads and writes it. If you want to use three pass, this means you will have to use –pass 1 for the first pass, –pass 3 for the second, and –pass 2 or 3 for the third.
rc_eq <string>
Ratecontrol equation. Recommended default: -rc_eq 'blurCplx^(1-qComp)'
qcomp <float>
QP curve compression: 0.0 => CBR, 1.0 => CQP. Recommended default: -qcomp 0.60
complexityblur <float>
Reduce fluctuations in QP (before curve compression) [20.0]
qblur <float>
Reduce fluctuations in QP (after curve compression) [0.5]
partitions <string>
One of H.264's most useful features is the ability to choose among many combinations of inter and intra partitions. P-macroblocks can be subdivided into 16×8, 8×16, 8×8, 4×8, 8×4, and 4×4 partitions. B-macroblocks can be divided into 16×8, 8×16, and 8×8 partitions. I-macroblocks can be divided into 4×4 or 8×8 partitions. Analyzing more partition options improves quality at the cost of speed. The default is to analyze all partitions except p4x4 (p8x8, i8x8, i4x4, b8x8), since p4x4 is not particularly useful except at high bitrates and lower resolutions. Note that i8x8 requires 8x8dct, and is therefore a High Profile-only partition. p8x8 is the most costly, speed-wise, of the partitions, but also gives the most benefit. Generally, whenever possible, all partition types except p4x4 should be used.
p8x8 (x264) /+partp8x8 (FFmpeg)
p4x4 (x264) /+partp4x4 (FFmpeg)
b8x8 (x264) /+partb8x8 (FFmpeg)
i8x8 (x264) /+parti8x8 (FFmpeg)
i4x4 (x264) /+parti4x4 (FFmpeg)
directpred <integer>
B-frames in H.264 can choose between spatial and temporal prediction mode. Auto allows x264 to pick the best of these; the heuristic used is whichever mode allows more skip macroblocks. Auto should generally be used.
Motion Prediction:
flags2 +wpred
This allows B-frames to use weighted prediction options other than the default. There is no real speed cost for this, so it should always be enabled.
dia (x264) / epzs (FFmpeg) is the simplest search, consisting of starting at the best predictor, checking the motion vectors at one pixel upwards, left, down, and to the right, picking the best, and repeating the process until it no longer finds any better motion vector.
hex (x264) / hex (FFmpeg) consists of a similar strategy, except it uses a range-2 search of 6 surrounding points, thus the name. It is considerably more efficient than DIA and hardly any slower, and therefore makes a good choice for general-use encoding.
umh (x264) / umh (FFmpeg) is considerably slower than HEX, but searches a complex multi-hexagon pattern in order to avoid missing harder-to-find motion vectors. Unlike HEX and DIA, the merange parameter directly controls UMH's search radius, allowing one to increase or decrease the size of the wide search.
esa (x264) / full (FFmpeg) is a highly optimized intelligent search of the entire motion search space within merange of the best predictor. It is mathematically equivalent to the bruteforce method of searching every single motion vector in that area, though faster. However, it is still considerably slower than UMH, with not too much benefit, so is not particularly useful for everyday encoding.
me_method <epzs,hex,umh,full>
One of the most important settings for x264, both speed and quality-wise.
me_range <integer>
MErange controls the max range of the motion search. For HEX and DIA, this is clamped to between 4 and 16, with a default of 16. For UMH and ESA, it can be increased beyond the default 16 to allow for a wider-range motion search, which is useful on HD footage and for high-motion footage. Note that for UMH and ESA, increasing MErange will significantly slow down encoding.
1: Fastest, but extremely low quality. Should be avoided except on first pass encoding.
6-7: 6 is the default. Activates rate-distortion optimization for partition decision. This can considerably improve efficiency, though it has a notable speed cost. 6 activates it in I/P frames, and subme7 activates it in B frames.
8-9: Activates rate-distortion refinement, which uses RDO to refine both motion vectors and intra prediction modes. Slower than subme 6, but again, more efficient.
subq
An extremely important encoding parameter which determines what algorithms are used for both subpixel motion searching and partition decision.
flags2 +mixed_refs
H.264 allows p8x8 blocks to select different references for each p8x8 block. This option allows this analysis to be done, and boosts quality with little speed impact. It should generally be used, though it obviously has no effect with only one reference frame.
flags2 +dct8x8
Gives a notable quality boost by allowing x264 to choose between 8×8 and 4×4 frequency transform size. Required for i8x8 partitions. Speed cost for this option is near-zero both for encoding and decoding; the only reason to disable it is when one needs support on a device not compatible with High Profile.
0: disabled
1: enabled only on the final encode of a MB
2: enabled on all mode decisions
trellis <0,1,2>
The main decision made in quantization is which coefficients to round up and which to round down. Trellis chooses the optimal rounding choices for the maximum rate-distortion score, to maximize PSNR relative to bitrate. This generally increases quality relative to bitrate by about 5% for a somewhat small speed cost. It should generally be enabled. Note that trellis requires CABAC.
flags2 -fastpskip
By default, x264 will skip macroblocks in P-frames that don't appear to have changed enough between two frames to justify encoding the difference. This considerably speeds up encoding. However, for a slight quality boost, P-skip can be disabled. In this case, the full analysis will be done on all P-blocks, and the only skips in the output stream will be the blocks whose motion vectors happen to match that of the skip vector and motion vectors happen to match that of the skip vector and which have no residual. The speed cost of enabling no-fast-pskip is relatively high, especially with many reference frames. There is a similar B-skip internal to x264, which is why B-frames generally encode much faster than P-frames, but it cannot be disabled on the commandline.