Unicorn

: A System for Searching the Social Graphby Facebook(7 total)

What we actually want is a way to force some fraction of the final results to possess a trait, while not requiring this trait from all results. This is what the WeakAnd operator accomplishes.

— Facebook, db.disi.unitn.eu

Facebook Search Logics

While each user has 130 friends on average, they have approximately 48000 friends-of-friends5. Even assuming only 4 bytes per posting list entry, this would still take over 178TB to store the posting lists for 1 billion users, which would require hundreds of machines.

— Facebook, research.fb.com

Understanding Queries NLU Unicorn Query Engine

Jon is also a friend-of-friends of himself since he is—by definition—a friend of his friends.

— Facebook, research.fb.com

Understanding Queries NLU Unicorn Query Engine

For storing per-entity metadata, Unicorn provides a forward index, which is simply a map of id to a blob that contains metadata for the id. The forward index for an index shard only contains entries for the ids that reside on that shard.

— Facebook, research.fb.com

Understanding Queries NLU Unicorn Query Engine

Unicorn also supports a Difference operator, which re-turns results from the first operand that are not present in the second operand. Continuing with the example above, we could find female friends of Jon Jones who are not friends of Lea Lin by using (difference (and friend:5 gender:1) friend:6). A…

— Facebook, research.fb.com

Understanding Queries NLU Unicorn Query Engine

Although there are many billions of nodes in the social graph, it is quite sparse: a typical node will have less than one thousand edges connecting it to other nodes. The average user has approximately 130 friends. The most popular pages and applications have tens of millions of edges, but these pag…

— Facebook, research.fb.com

Understanding Queries NLU Unicorn Query Engine

Unicorn is designed to answer billions of queries per day at latencies in the hundreds of milliseconds, and it serves as an infrastructural building block for Facebook’s Graph Search product.

— Facebook, research.fb.com

Understanding Queries NLU

W W WW WW W MW W WW W W WW W W W WW W W &W W WW W W WW &W W W& WW WW& W W K W WW& WW W W M W W MWKW &WW W & W WW WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWMWWMMMMMMMWMMMMMMMMW WM&&&MMWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWMMK KWWWWWWWWWWWWWWWWWWWWWWWM KW WWWWWWWWWWWWWWWWMMMMMMMMMMMMWWWWWWWWWMMMMM&&&MM&&&&&&&&&&&&&&&&&&&&&&&&&&&&W WKKKKKK&&MWWWWWMMMMMM&&&&&WWWWWWWWWWWWWM&&K &WWWWWWWWWWWWWWWMM WWKK WWWWWWWWWWWWWWWWWWWMMMMMMWWW MWWMMM&&MMMMMM&&&&M&&&&&&&&&&&&&&&&&&&W WKKKKKK&MMWWWWWMMMMMMM&&&MMWWWWWWWWWWKWWM&K &WWWWWWWWWWWWWWWWW WK WWWWWWWWWWWWWWWWWWWWWWWWWW KWMMMMM&MMMMMMMMMM&&MM&MMMMMM&&MMMMMW W&KKKKK&MMWWWWWMMMMMMM&&&&MWWWWWWW& &&K MWWWWWWWWWWWWWWWWWWWM W& WWWWWWWWWWWWWWWWWWWWWWWW WWMMMMMMMMMMMMMMMWWWWWM&WWWWMMMMMMW W&KKKKK&MMWWWWWWMMMM&M&&&&MWWWWWWW WK MWWWWWWWWWWWWWWWWWMM& W& WWWWWWWWWWWWWWWWWWWWWWWW WWMMMMM&MMMMMMWW MWWMMMWW W&&KKK&&MMWWWWWMMMMMMMM&&&MWWWWWWW W MK MWWWWWWWWWWWWWWWKKK W& WWWWWWWWWWWWWWWWWWWWWWW WMMMMMMMMMMWM WWWWWW W&&KKKK&MMWWWWWWWMMMMMM&&&MWWWWWWW K WK &WWWWWWWWWWWWWWMMKK W& WWWWWWWWWWWWWWWWWWWWWWW WMMMMMMMMMW& WWWWW W&&&&&&&MMWWWWWWWWMMMMMM&MMWWWWWWW WK &WWWWWWWWWWWWWWW&KKK W& WWWWWWWWWWWWWWWWWWWWWWW KWMMMMMMMM&K WWWW W&&&&&&&MMWWWWWWWMMMMMMM&&MWWWWWWW WK &WWWWWWWWWWWWWWWM&KK W& WWWWWWWWWWWWWWWWWWWWWWW WWWMMMMMMM& WWW W&K&&&&&MMWWWWWWWMMMMMM&&&MWWWWWWK WK &WWWWWWWWWWWWWWWMKK W& WWWWWWWWWWWWWWWWWWWWWWW WWWMMMMMM&& KK MWW W&K&&&&&MMWWWWWWWWMMMMM&&&MWWWWWW KWK MWWWWWWWWWWWWWWM&KK &WK WWWWWWWWWWWWWWWWWWWWWWW K WWWWMMMMMM& WW W&&&&&&&MMWWWWWWWMMMMMMM&&MWWWWWW MM& MWWWWWWWWWWWWWWWWWWWWWWW& K WWWWWWWWWWWWWWWWWWWWWWWW WWWWMMMMMMMW W W&&&&&&&MMWWWWWWWMMMMMM&&&MWWWWWW WM& MWWWWWWWWWWWWWWMKKK & WWWWWWWWWWWWWWWWWWWWWWWWWM MWWWWMMWMMMMM W W&&&&&&&MMWWWWWMMMMMMM&&&&&WWWWWW W&K KMWWWWWWWWWWWWWWM&KK K WWWWWWWWWWWWWW KKWWWWWWM W&&&&&&&&&&&K K W&&&&&&&&MWWWWWWMMMMM&&&KK&WWWWWW W&KWM& KMWWWWWWWWWWWWWWM&KK KWMMMMMMMMMMM WWWWWWW WKK&&&&&&&MWWWWMMMM&&&&KKK&WWWWWWW&WWW WWWM& KMWWWWWWWWWWWWWWM&KK K WWWWWWWWWWWM WWWWWWWM W KK&&&&&&MWWWWMM&&&&&&KKKKMWWWWW&WWWWW&MMMK MWWWWWWWWWWWWWMM&KK K K&WWWWWWWWW WM MWKK WKKKKKKKK&MMWWMM&&&&&&KK KMWWWWWMWWWWW&W&& MWWWWWWWWWWWWWM&KKK KMWWWWWWWW W KKKKKKK&&MMMM&&&KK&KKK MWWWWWMWWWWW&MK &MWWWWWWMWWWWW&KKK WKKKKKKK&&&MMMM&&KKKKKK &MWWWWKWWWWWMK &MWWWWWMWWWWWW&KK WKKKKK KK&&&MM&&KKK K&&&&&&WWWWWW &MWWWWWMMWWWWW&KKK WK KK KKKK&&&&K KK WWW &MWWWWMMMMWWWW&&KKK WK K KKK&KK WWW KMMWWMMMMMWWWWMM&K WK K KKK WWW K&MMMMM&MMWWWWM K WK K WWW K&MMMMM&&MWWM WK K WWW K&MMMM&&&MM& WK K WWW K&MM&&&&&&& & WK KWK WW& K&MM&&K&&& WK K &WWWWWWWWW&&KWWW W M KK&M&KKKK

, , : . , , . , : , . :,, . .. , :: . . :, , , . . : . .., ,.,,,,,::,,. . , ,:,: :,,::,, . ., . .,: ,,::,: . :., . .,, ,,::, ,: .,: , , ,,. .,,, , :, .., ,:. . , , . , : ., , , .: ., ,,:,,: :: : .,. :,,,., ,: :,,.,, ,.: : ,,,., , , : . : . .,.. , , . , ,., .,, , ,. . ,.... : : .,,. , : , . , . : ..........: ,.,..,,. , ,., . .. : .,,. ...,.....,,.: ,., ,,,. :: ,. . ,. : : ....,,..,....,:::.....,, :,. ,.... : . .. .........,,.,...,,,....,..,,, :,,.,. , :: :. : :: ,.. . ..........,,,,.....,,....,.,.. : :.,,, .. . :: , . .. ............,,,,,... ....,..,,:,:... ,:.... : ,. , .............,,,,,,. .,,,,:,,,:,:,,,,,...... ..,.... : ............,,,,,,,..,:.,..,.:::...,,,,....,,,... .. ...........,,,,,,,,..:.,,..,:.,..,,,,,.,..,,.,,....,

} ~* < <~ < ~ < ~ <~ ~ * ~ < ~ < } } < ~} ~ } * } } }}}}}}}}}}}}}}}}}}}}}}}}}}} } }}}}}}}}}}}}}}}}}}}}}}}}}}}* ****}}} }}}}}}}}}}}} }}}**<<<*} }}****<<*** * < }<~ ~ * ~ ~ } } } < * }*~ < *~~~ } * ~* * * } * ~ < < <~ ~ ~ ~ } < * <} ~ }~ * } }}}} }}} ~ ~ } * < < }}*} * < ~ ~ < < } ~ *} *} * < < < ~ } } <*~~* **~ << < <* ~ }*} } }~* ~ *~ }<*** } ~ * }}*} } * ~ } } } ~ } }} * *} } <<* * }<<<<<* ~ < } <~<<< }* }}}} } }*** ~*} * * }~ } *< }}}* ~<<< ~<**<<~ }} ******** ~ * < } ~~ **<*}}*~ ** <<******< ~ ~ *}}}}}}~ }** }~ <<*<<<<< <<} }}}} } } }*<* }}*< <~<<<<~ ~ }}< ~* }} * * }}*<* }<~ ~~~ } ~ ~ ~ < <~ <*}*~ }*}}}* * }}**<* }<<~ ~<<~~ ~~*<*~ * }*}}}* *}}}*<<* }* *<~~<***< }<<< * }}}**}} }*<<} * *}***<<* }}***<****< * ~ < } }* ~* <<<* }*} *} }}}***< <} } } }}* }~ <<<* }***} }* }*} }*<

U V U V U o U U o o U U j ooooojjjjjjjj j j jjjjjj jjooooojjjjj jjjooooojjjjoj V U UoV U UVjoU UUVooU V V UojoU UVVoooo U UVjoU ooVV U U UVjoU ooVV U jj V UVjoU UVVoVo U o VUUUUVV j UVjoU UVVoVV U j oUUUUUUVVoo j UVjoU UVoVo U jjj jUUUVVVV VVVVj UojoU UVooo U oj Vo j VVVVUUVo j j o UojoU UUVoVo U o VUUooojU j joVUU UVo V o UojoU UVojo U UjoVVU UVoU VooU V V jVUUVj j o j UVjoU U U UooVVj VVjVUVo o Vo U UVooU UVojoVU V VVVVVo Vj o j jjoVVojoUj U V VoV UVVooVU o VVVU UU joj U jVVUjoVoU UVj U VVU o VoV UVVooVVUo VVUUUVU UUUo j oVVVVVVVoVVVoo jVV jV oVoj j UoV UVooooV V U UVoVoo j oVVoo VUVVVVVVVVVVVoj joj jjjjj U VoVU UVoojjV o o UVj U V jVUUVVVVVVoVo j jo UU UoooU UVojjoVUo oU VUV VVo oVoj j oVoooj jjjVVo U UUU UVjjoU UVojjjoVUj oVoojjjjjoo jUUjoVVUUUU oooVjj UUVU VVo oU UUVoojjoVUj j UUVjVVVUV jUooVV U UUUUUVVV Vooj jV UVVojjoVUj UUVooUVV U UU UUUUVVVVVVVU UUVVj jjjjj jV UVjjjoj VVVojoj jj U UUUUU UUVVVVVoooVVVoooooo jjj jV UVj oUVj oVUUVoj U UUUUUUU UVVooooooooooooj jj jj jV V j UUVoo U UUUUUUVUUVVooojjjjjjooooj j jV V VUUUoo U UUUUUVVVVVoojjjjjjjjjoooj jV V jj j VVUUVo U UUUUVVVoooojj jjjj j jV Uojjj jV UVoVUo UUUUVVVooojjjj jVooVoooVVVVU o jo jV Uj jjj j VUVj jUVVVUo U UUUVVojj j o oo VU jV Vj jj j j

& & K K K W MWWWWWWWWWWWM&WWWWWWWWWWWWWWWW WWWWWWWWWWWWWWWWW MWWWWWWWWMWK MWWWWWWWWW KWWWWWWWWMMWWW W&MWWWWWMMWWW M &WWWWWWWW WK MWWWWWWWW && &WWWW &W WMMWWWWWMMWWW W KWWWWWWM& WK WWWWWWWWW K WWWW && W WMMWWWWWWMWWW W &WWWWWWM& WK WWWWWWWWW& KK WWWW WMMWWWWWMMWWW M &WWWWWWWW&W WWWWWWWWWWWK KWWWWW KK WMMWWWWMM&WWW KW &WWWWWWM& K KWWWW& KWWW M&KKK MMMWWWM&&KWWWWWWM &WWWWWWM& K &&M&& & M&&MWM&&&KMWWWWM &WWWWWM&K K && M&&MM&KK KK WW KWWWWWWW& MK M&&K&K WW KWWWWW MM M&K WM KWWMW & MM M& WWWWWWW MM KWWM &MW& KK K&WWWM& WW KWMK MM& KW MWWWWWWWWWW KMM &K KM &WWWWWWW& & KW& K& WWWWW&KMMWWM &W M K W &K W K K WWWWWW& W K WM WK & MWK KKW&M WWW & K&WMWWM WWWMMWMMWMWK KWWWWM WWWWW&WMK &MWWM& &MMWW& KMWWMMK &&WK WWWWWW&

, : , : ,. . .. , , : , . , , . . , .. ,.., : ..,,,.,... ,,. : ..,,,,,,..,,::. ,. ..,,,,,,,,,,,:.,.,,,.. ..,,,,,:,,.,.,::,,.. ,, ....,,,,,,,.,:,.,,,.. ...,,,:,:,,,:,, ::,., .,,,,,:::,,,,. .,,,,,:,:,,,,. : , ,,,,,,,,,,,.,. :: :::: ::::,,,. : . :., , :: : :: , : ., :: : :,,: , :: :::,. : :, .. ::: ::, , ,, : : ::. ::,: : . ::: :

* }* ~ * } ~ } } ~ ***<<<~~<<<<<<<<<<<<<<<<<<<<< ~~<<<<~~~<<*<<<<~ ~<*<<*<<~~< < <~* ~ } } < } < ~ * * ~ * } } ~* } *} < ~<~ ~ < **< < < < }~< } ~ * } ~ } } * } < * * }** * < <~~~ *<* } < } }*} <} }}}}<}} ~~* }}*} < ~*<*~ <*<<*~ < * }} } <~}* <~* <}**< << **~~< } < * <~** }*~ *}}<<~ ~ *< <*~~< * * < < ~ } **<~ <}*~~~ ~ ~~~~ ~~ *<<<***< < * }* }}}}}*~ *}<~~~~~ ~ ~~~ ~<<<<~<<<~ }}<* }*< }** ~ **<~~~~~ ~~~~ <*}}*~ < * }*< } } < *}<<~~~~ ~~~~ ~<*}**~ <<* * ~~ ~ }* < *}~~~~~~~ <~~~~~ ~<<*<~~~<<*~<* ~ * } } ~< < <<<<~~~<}~<<<<~~~<<<**~<~<<< <<~ <<~~< } ~

o V U oUV o U Vo o V VV jo U jj V V U o j U oj V V o Uo VV U j V o V o UVoU oUUj Uj V V U ojU VU V j jj U V V o VVoo VooUo o U V V o VU VV VV jo oo V Uo V V j V V jVV V UU U Voj VV jUj V VVVUVjj jjj U o V UUVjjooVjo j j UVojjj U V j V Uoj jj UooUVVV o Vj Vj U j jojVooj V j Uj j VjojoVoj V o o UU j jjjj ooj V j j j ojjj j Vo UU VV V V oj jjjjjj V U UV Uoojjjj UV o j Uoj oojjjj oo jj jV U Vj V o U VUUo j jU V j j U o U o U o j j j V ooo o