Dividing data into testing, training, developing sets. Crawling for test data Loading time: 0.005738735198974609 Now testing on 2500 samples. Finished with ././GroundTruth/TW/tgl/tl.Test.txt Finished with ././GroundTruth/TW/spa/es.Test.txt Finished with ././GroundTruth/TW/ita/it.Test.txt Finished with ././GroundTruth/TW/tha/th.Test.txt Finished with ././GroundTruth/TW/est/et.Test.txt Finished with ././GroundTruth/TW/hin/hi.Test.txt Finished with ././GroundTruth/TW/eng/en.Test.txt Finished with ././GroundTruth/TW/ukr/uk.Test.txt Finished with ././GroundTruth/TW/ell/el.Test.txt Finished with ././GroundTruth/TW/fra/fr.Test.txt Finished with ././GroundTruth/TW/por/pt.Test.txt Finished with ././GroundTruth/TW/ara/ar.Test.txt Finished with ././GroundTruth/TW/rus/ru.Test.txt Finished with ././GroundTruth/TW/nld/nl.Test.txt Finished with ././GroundTruth/TW/hat/ht.Test.txt Finished with ././GroundTruth/TW/deu/de.Test.txt Finished with ././GroundTruth/TW/kor/ko.Test.txt Finished with ././GroundTruth/TW/zho/zh.Test.txt Finished with ././GroundTruth/TW/ind/in.Test.txt Finished with ././GroundTruth/TW/jpn/ja.Test.txt Finished with ././GroundTruth/TW/fin/fi.Test.txt Finished with ././GroundTruth/TW/tur/tr.Test.txt Finished with ././GroundTruth/TW/pol/pl.Test.txt Finished with ././GroundTruth/TW/swe/sv.Test.txt Finished with ././GroundTruth/TW/ron/ro.Test.txt Finished with all files. CLD2 ----------------------- ---CLASSIFIER REPORT--- ----------------------- precision recall f1-score support afr 0.00 0.00 0.00 0 ara 1.00 0.89 0.94 1717 aze 0.00 0.00 0.00 0 cat 0.00 0.00 0.00 0 dan 0.00 0.00 0.00 0 deu 1.00 0.91 0.95 143 ell 0.97 1.00 0.98 30 eng 0.98 0.99 0.98 10377 est 1.00 0.05 0.09 21 fin 1.00 0.68 0.81 19 fra 0.99 0.93 0.96 506 glg 0.00 0.00 0.00 0 hat 1.00 0.17 0.29 24 hin 1.00 0.39 0.56 31 ind 1.00 0.73 0.84 2090 ita 0.98 0.91 0.94 248 jpn 1.00 0.99 0.99 2009 kin 0.00 0.00 0.00 0 kor 0.99 0.98 0.99 121 mkd 0.00 0.00 0.00 0 mlt 0.00 0.00 0.00 0 msa 0.00 0.00 0.00 0 nld 1.00 0.89 0.94 125 pol 1.00 0.90 0.95 49 por 0.99 0.93 0.96 1409 ron 0.80 0.47 0.59 17 rus 0.99 0.95 0.97 518 spa 1.00 0.91 0.95 3518 sqi 0.00 0.00 0.00 0 srp 0.00 0.00 0.00 0 sun 0.00 0.00 0.00 0 swa 0.00 0.00 0.00 0 swe 1.00 0.93 0.97 30 tgl 1.00 0.65 0.79 208 tha 0.98 1.00 0.99 135 tur 1.00 0.96 0.98 425 ukr 0.91 0.71 0.80 14 unk 0.00 0.00 0.00 0 uzb 0.00 0.00 0.00 0 zho 1.00 0.29 0.44 7 avg / total 0.99 0.93 0.96 23791 LangID ----------------------- ---CLASSIFIER REPORT--- ----------------------- precision recall f1-score support afr 0.00 0.00 0.00 0 amh 0.00 0.00 0.00 0 ara 1.00 0.93 0.96 1717 arg 0.00 0.00 0.00 0 asm 0.00 0.00 0.00 0 aze 0.00 0.00 0.00 0 bel 0.00 0.00 0.00 0 bos 0.00 0.00 0.00 0 bre 0.00 0.00 0.00 0 bul 0.00 0.00 0.00 0 cat 0.00 0.00 0.00 0 ces 0.00 0.00 0.00 0 cym 0.00 0.00 0.00 0 dan 0.00 0.00 0.00 0 deu 0.81 0.97 0.88 143 ell 1.00 1.00 1.00 30 eng 0.98 0.94 0.96 10377 epo 0.00 0.00 0.00 0 est 0.00 0.00 0.00 21 eus 0.00 0.00 0.00 0 fao 0.00 0.00 0.00 0 fas 0.00 0.00 0.00 0 fin 0.64 0.84 0.73 19 fra 0.90 0.95 0.93 506 gle 0.00 0.00 0.00 0 glg 0.00 0.00 0.00 0 hat 0.44 0.17 0.24 24 heb 0.00 0.00 0.00 0 hin 0.92 0.39 0.55 31 hrv 0.00 0.00 0.00 0 hun 0.00 0.00 0.00 0 hye 0.00 0.00 0.00 0 ind 0.97 0.58 0.73 2090 isl 0.00 0.00 0.00 0 ita 0.81 0.90 0.85 248 jav 0.00 0.00 0.00 0 jpn 0.99 0.99 0.99 2009 kat 0.00 0.00 0.00 0 kaz 0.00 0.00 0.00 0 khm 0.00 0.00 0.00 0 kin 0.00 0.00 0.00 0 kir 0.00 0.00 0.00 0 kor 0.85 0.98 0.91 121 kur 0.00 0.00 0.00 0 lao 0.00 0.00 0.00 0 lat 0.00 0.00 0.00 0 lav 0.00 0.00 0.00 0 lit 0.00 0.00 0.00 0 ltz 0.00 0.00 0.00 0 mar 0.00 0.00 0.00 0 mkd 0.00 0.00 0.00 0 mlg 0.00 0.00 0.00 0 mlt 0.00 0.00 0.00 0 mon 0.00 0.00 0.00 0 msa 0.00 0.00 0.00 0 nep 0.00 0.00 0.00 0 nld 0.87 0.94 0.90 125 nno 0.00 0.00 0.00 0 nob 0.00 0.00 0.00 0 nor 0.00 0.00 0.00 0 oci 0.00 0.00 0.00 0 ori 0.00 0.00 0.00 0 pan 0.00 0.00 0.00 0 pol 0.68 0.96 0.80 49 por 0.98 0.92 0.95 1409 pus 0.00 0.00 0.00 0 que 0.00 0.00 0.00 0 ron 0.67 0.47 0.55 17 rus 1.00 0.95 0.97 518 sin 0.00 0.00 0.00 0 slk 0.00 0.00 0.00 0 slv 0.00 0.00 0.00 0 sme 0.00 0.00 0.00 0 spa 0.98 0.94 0.96 3518 sqi 0.00 0.00 0.00 0 srp 0.00 0.00 0.00 0 swa 0.00 0.00 0.00 0 swe 0.83 1.00 0.91 30 tam 0.00 0.00 0.00 0 tgl 0.69 0.57 0.62 208 tha 0.97 1.00 0.99 135 tur 0.99 0.88 0.93 425 uig 0.00 0.00 0.00 0 ukr 0.67 0.71 0.69 14 urd 0.00 0.00 0.00 0 vie 0.00 0.00 0.00 0 vol 0.00 0.00 0.00 0 wln 0.00 0.00 0.00 0 xho 0.00 0.00 0.00 0 zho 0.07 0.86 0.12 7 zul 0.00 0.00 0.00 0 avg / total 0.97 0.91 0.93 23791 LangDetect ----------------------- ---CLASSIFIER REPORT--- ----------------------- precision recall f1-score support afr 0.00 0.00 0.00 0 ara 1.00 0.98 0.99 1717 bul 0.00 0.00 0.00 0 cat 0.00 0.00 0.00 0 ces 0.00 0.00 0.00 0 cym 0.00 0.00 0.00 0 dan 0.00 0.00 0.00 0 deu 0.56 0.95 0.71 143 ell 1.00 1.00 1.00 30 eng 0.98 0.98 0.98 10377 est 0.04 0.05 0.05 21 fas 0.00 0.00 0.00 0 fin 0.50 0.84 0.63 19 fra 0.96 0.94 0.95 506 hat 0.00 0.00 0.00 24 hin 1.00 0.39 0.56 31 hrv 0.00 0.00 0.00 0 hun 0.00 0.00 0.00 0 ind 0.97 0.92 0.95 2090 ita 0.84 0.91 0.87 248 jpn 1.00 0.98 0.99 2009 kor 0.96 0.92 0.94 121 lit 0.00 0.00 0.00 0 mkd 0.00 0.00 0.00 0 nld 0.83 0.90 0.86 125 nor 0.00 0.00 0.00 0 pol 0.86 0.88 0.87 49 por 0.92 0.93 0.93 1409 ron 0.64 0.53 0.58 17 rus 0.99 0.97 0.98 518 slk 0.00 0.00 0.00 0 slv 0.00 0.00 0.00 0 som 0.00 0.00 0.00 0 spa 0.99 0.91 0.95 3518 sqi 0.00 0.00 0.00 0 swa 0.00 0.00 0.00 0 swe 0.83 1.00 0.91 30 tgl 0.72 0.74 0.73 208 tha 0.99 1.00 1.00 135 tur 1.00 0.96 0.98 425 ukr 0.91 0.71 0.80 14 unk 0.00 0.00 0.00 0 urd 0.00 0.00 0.00 0 vie 0.00 0.00 0.00 0 zho 0.00 0.00 0.00 7 avg / total 0.97 0.95 0.96 23791 DOMAIN RESULTS cld2 TW ----------------------- ---CLASSIFIER REPORT--- ----------------------- precision recall f1-score support afr 0.00 0.00 0.00 0 ara 1.00 0.89 0.94 1717 aze 0.00 0.00 0.00 0 cat 0.00 0.00 0.00 0 dan 0.00 0.00 0.00 0 deu 1.00 0.91 0.95 143 ell 0.97 1.00 0.98 30 eng 0.98 0.99 0.98 10377 est 1.00 0.05 0.09 21 fin 1.00 0.68 0.81 19 fra 0.99 0.93 0.96 506 glg 0.00 0.00 0.00 0 hat 1.00 0.17 0.29 24 hin 1.00 0.39 0.56 31 ind 1.00 0.73 0.84 2090 ita 0.98 0.91 0.94 248 jpn 1.00 0.99 0.99 2009 kin 0.00 0.00 0.00 0 kor 0.99 0.98 0.99 121 mkd 0.00 0.00 0.00 0 mlt 0.00 0.00 0.00 0 msa 0.00 0.00 0.00 0 nld 1.00 0.89 0.94 125 pol 1.00 0.90 0.95 49 por 0.99 0.93 0.96 1409 ron 0.80 0.47 0.59 17 rus 0.99 0.95 0.97 518 spa 1.00 0.91 0.95 3518 sqi 0.00 0.00 0.00 0 srp 0.00 0.00 0.00 0 sun 0.00 0.00 0.00 0 swa 0.00 0.00 0.00 0 swe 1.00 0.93 0.97 30 tgl 1.00 0.65 0.79 208 tha 0.98 1.00 0.99 135 tur 1.00 0.96 0.98 425 ukr 0.91 0.71 0.80 14 unk 0.00 0.00 0.00 0 uzb 0.00 0.00 0.00 0 zho 1.00 0.29 0.44 7 avg / total 0.99 0.93 0.96 23791 DOMAIN RESULTS langid TW ----------------------- ---CLASSIFIER REPORT--- ----------------------- precision recall f1-score support afr 0.00 0.00 0.00 0 amh 0.00 0.00 0.00 0 ara 1.00 0.93 0.96 1717 arg 0.00 0.00 0.00 0 asm 0.00 0.00 0.00 0 aze 0.00 0.00 0.00 0 bel 0.00 0.00 0.00 0 bos 0.00 0.00 0.00 0 bre 0.00 0.00 0.00 0 bul 0.00 0.00 0.00 0 cat 0.00 0.00 0.00 0 ces 0.00 0.00 0.00 0 cym 0.00 0.00 0.00 0 dan 0.00 0.00 0.00 0 deu 0.81 0.97 0.88 143 ell 1.00 1.00 1.00 30 eng 0.98 0.94 0.96 10377 epo 0.00 0.00 0.00 0 est 0.00 0.00 0.00 21 eus 0.00 0.00 0.00 0 fao 0.00 0.00 0.00 0 fas 0.00 0.00 0.00 0 fin 0.64 0.84 0.73 19 fra 0.90 0.95 0.93 506 gle 0.00 0.00 0.00 0 glg 0.00 0.00 0.00 0 hat 0.44 0.17 0.24 24 heb 0.00 0.00 0.00 0 hin 0.92 0.39 0.55 31 hrv 0.00 0.00 0.00 0 hun 0.00 0.00 0.00 0 hye 0.00 0.00 0.00 0 ind 0.97 0.58 0.73 2090 isl 0.00 0.00 0.00 0 ita 0.81 0.90 0.85 248 jav 0.00 0.00 0.00 0 jpn 0.99 0.99 0.99 2009 kat 0.00 0.00 0.00 0 kaz 0.00 0.00 0.00 0 khm 0.00 0.00 0.00 0 kin 0.00 0.00 0.00 0 kir 0.00 0.00 0.00 0 kor 0.85 0.98 0.91 121 kur 0.00 0.00 0.00 0 lao 0.00 0.00 0.00 0 lat 0.00 0.00 0.00 0 lav 0.00 0.00 0.00 0 lit 0.00 0.00 0.00 0 ltz 0.00 0.00 0.00 0 mar 0.00 0.00 0.00 0 mkd 0.00 0.00 0.00 0 mlg 0.00 0.00 0.00 0 mlt 0.00 0.00 0.00 0 mon 0.00 0.00 0.00 0 msa 0.00 0.00 0.00 0 nep 0.00 0.00 0.00 0 nld 0.87 0.94 0.90 125 nno 0.00 0.00 0.00 0 nob 0.00 0.00 0.00 0 nor 0.00 0.00 0.00 0 oci 0.00 0.00 0.00 0 ori 0.00 0.00 0.00 0 pan 0.00 0.00 0.00 0 pol 0.68 0.96 0.80 49 por 0.98 0.92 0.95 1409 pus 0.00 0.00 0.00 0 que 0.00 0.00 0.00 0 ron 0.67 0.47 0.55 17 rus 1.00 0.95 0.97 518 sin 0.00 0.00 0.00 0 slk 0.00 0.00 0.00 0 slv 0.00 0.00 0.00 0 sme 0.00 0.00 0.00 0 spa 0.98 0.94 0.96 3518 sqi 0.00 0.00 0.00 0 srp 0.00 0.00 0.00 0 swa 0.00 0.00 0.00 0 swe 0.83 1.00 0.91 30 tam 0.00 0.00 0.00 0 tgl 0.69 0.57 0.62 208 tha 0.97 1.00 0.99 135 tur 0.99 0.88 0.93 425 uig 0.00 0.00 0.00 0 ukr 0.67 0.71 0.69 14 urd 0.00 0.00 0.00 0 vie 0.00 0.00 0.00 0 vol 0.00 0.00 0.00 0 wln 0.00 0.00 0.00 0 xho 0.00 0.00 0.00 0 zho 0.07 0.86 0.12 7 zul 0.00 0.00 0.00 0 avg / total 0.97 0.91 0.93 23791 DOMAIN RESULTS langdetect TW ----------------------- ---CLASSIFIER REPORT--- ----------------------- precision recall f1-score support afr 0.00 0.00 0.00 0 ara 1.00 0.98 0.99 1717 bul 0.00 0.00 0.00 0 cat 0.00 0.00 0.00 0 ces 0.00 0.00 0.00 0 cym 0.00 0.00 0.00 0 dan 0.00 0.00 0.00 0 deu 0.56 0.95 0.71 143 ell 1.00 1.00 1.00 30 eng 0.98 0.98 0.98 10377 est 0.04 0.05 0.05 21 fas 0.00 0.00 0.00 0 fin 0.50 0.84 0.63 19 fra 0.96 0.94 0.95 506 hat 0.00 0.00 0.00 24 hin 1.00 0.39 0.56 31 hrv 0.00 0.00 0.00 0 hun 0.00 0.00 0.00 0 ind 0.97 0.92 0.95 2090 ita 0.84 0.91 0.87 248 jpn 1.00 0.98 0.99 2009 kor 0.96 0.92 0.94 121 lit 0.00 0.00 0.00 0 mkd 0.00 0.00 0.00 0 nld 0.83 0.90 0.86 125 nor 0.00 0.00 0.00 0 pol 0.86 0.88 0.87 49 por 0.92 0.93 0.93 1409 ron 0.64 0.53 0.58 17 rus 0.99 0.97 0.98 518 slk 0.00 0.00 0.00 0 slv 0.00 0.00 0.00 0 som 0.00 0.00 0.00 0 spa 0.99 0.91 0.95 3518 sqi 0.00 0.00 0.00 0 swa 0.00 0.00 0.00 0 swe 0.83 1.00 0.91 30 tgl 0.72 0.74 0.73 208 tha 0.99 1.00 1.00 135 tur 1.00 0.96 0.98 425 ukr 0.91 0.71 0.80 14 unk 0.00 0.00 0.00 0 urd 0.00 0.00 0.00 0 vie 0.00 0.00 0.00 0 zho 0.00 0.00 0.00 7 avg / total 0.97 0.95 0.96 23791