Navigacija
Lista poslednjih: 16, 32, 64, 128 poruka.

Import iz velikog .txt-a u mysql - više pitanja

[es] :: PHP :: Import iz velikog .txt-a u mysql - više pitanja

[ Pregleda: 1550 | Odgovora: 12 ]

Postavi temu Odgovori

Autor

Pretraga teme: Traži
Markiranje Štampanje RSS

asyk

Član broj: 46486
Poruke: 55
*.adsl.net.t-com.hr.



Profil

icon Import iz velikog .txt-a u mysql - više pitanja16.02.2007. u 09:06

Pozdrav svima.

Evo, na poslu mi šef dao zadatak da napravim import jednog velikog txt fajla u mysql bazu. E sad, imam nekoliko pitanja:

Prvo: txt file je velik oko 65 MB! Da li je uopce moguce importati ovoliko veliku datoteku, buduci da sam procitao http://www.elitesecurity.org/t125053 i shvatio da moram paziti koliko mi je velika datoteka ako koristim f-ju file()?

Drugo:

Zapisi u txt fajlu su formatirani (katastrofalno) na ovaj nacin:

Code:

1000039 PONTIN drustvo s ogranicenom odgovornoscu za ribarstvo,preradu ribe, trgovinu, turizam i usluge     PONTIN D.O.O.     0600888913 1999070613Zadarska  5207 Zadar  71951 Zadar  23000Gazenicka Cesta  32 5190019950217 999


Želim napraviti insert podataka u bazu na slijedeci nacin (po redu pojavljivanja):

1000039 - insert into column 1
PONTIN drustvo s ogranicenom odgovornoscu za ribarstvo,preradu ribe, trgovinu, turizam i usluge - insert into column 2
PONTIN D.O.O. - insert into column 3
0600888913 - insert into column 4
19990706 - insert into column 5
13 - insert into column 6
Zadarska - insert into column 7
5207 - insert into column 8
Zadar - insert into column 9
71951 - insert into column 10
Zadar - insert into column 10
23000 - insert into column 11
Gazenicka Cesta - insert into column 12
32 - insert into column 13
5190019950217 - insert into column 14
999 - insert into column 15

Koji pristup trebam zauzeti da riješim ovaj problem uspješno? Svaka pomoć bi mi zbilja dobro došla.

Unaprijed zahvaljujem na bilo kakvom korisnom savjetu.
16.02.2007. u 09:06 

japan

Član broj: 34328
Poruke: 368
*.BVCOM.NET.



Profil

icon Re: Import iz velikog .txt-a u mysql - više pitanja16.02.2007. u 09:42
nisam siguran sta je delimiter polja u ovom primeru koji si dao, ali mi je jedino logicno da je to \t, pa bi ti kod onda izgledao ovako nekako:
Code:
LOAD DATA INFILE 'importfile.txt'
INTO TABLE table_name
FIELDS TERMINATED BY '\t'
LINES TERMINATED BY '\n'
(field1, filed2, field3, ...)


http://dev.mysql.com/doc/refman/5.0/en/load-data.html


ako nisam u pravu za ovaj \t pozdravi majstora koji je formatirao csv...
16.02.2007. u 09:42 

asyk

Član broj: 46486
Poruke: 55
*.adsl.net.t-com.hr.



Profil

icon Re: Import iz velikog .txt-a u mysql - više pitanja16.02.2007. u 11:59
"\t" je koliko sam shvatio Google "tab" delimiter? Ako sam u pravu, onda te moram razočarati i reći da podaci _nisu_ tab delimited, nego s običnim (i to jednim) spaceom. Ono što narod kaže "jedan zarez jebe cijeli tekst" :-)

Inače, ovaj link na LOAD DATA mi je vrlo, vrlo koristan, uopće nisam znao za njega. Al svejedno nemam pojma šta da sad radim, jer nije postavljen klasičan delimiter..
16.02.2007. u 11:59 

Nemanja Avramović
PHP developer, Webinsane
Mladenovac, Srbija

Moderator
Član broj: 32202
Poruke: 3762
212.200.145.*

ICQ: 266136396
Sajt: www.avramovic.info


Profil

icon Re: Import iz velikog .txt-a u mysql - više pitanja16.02.2007. u 12:30
Onda ćeš teško bilo šta uraditi, jer ako staviš razmak kao delimiter onda će svaku reč posebno odvajati... a to je loše... mislim, kako SQL da zna tačno gde da podeli liniju na slogove ako nema klasičnog delimitera?

Što reče "japan":
Citat:
pozdravi majstora koji je formatirao csv

Moj sajt. Moj blog. Moj avatar. Moj grad. Moja frizura.

[NE PRUŽAM PODRŠKU ZA PHP PREKO PRIVATNIH PORUKA!]
16.02.2007. u 12:30 

asyk

Član broj: 46486
Poruke: 55
*.adsl.net.t-com.hr.



Profil

icon Re: Import iz velikog .txt-a u mysql - više pitanja16.02.2007. u 13:34
Da stvar bude tužnija, ne radi se o "majstoru" nego o službenoj državnoj instituciji! Da ne pričam o tome koji su pothvati trebali biti poduzeti da se uopće dođe do ikakvog elektronskog zapisa ovako nečeg... užas.

Razmišljam o tome kako bi se sve te puste gluposti mogle nekako razgraditi preko regular expressionsa al da ide - ne ide! :~(
16.02.2007. u 13:34 

japan

Član broj: 34328
Poruke: 368
*.bvcom.net.



Profil

icon Re: Import iz velikog .txt-a u mysql - više pitanja16.02.2007. u 13:48
čini mi se da ti ne gine fizikalija, samo treba da vidiš koji pristup ti je najbezbolniji (ja bih probao da na neki način dovedem taj fajl u neko ispravno stanje) i da eventualno napraviš sebi neki alat za ovu avanturu.

u svakom slučaju, u 65M stane mnogo redova... pa srećno...
16.02.2007. u 13:48 

Whois

Član broj: 92214
Poruke: 32
77.46.130.*



Profil

icon Re: Import iz velikog .txt-a u mysql - više pitanja16.02.2007. u 14:16
Ako malo bolje pogledas, videces da su podaci tacno odredjeni delimiterima i vrlo lako ih je moguce staviti u mysql. Delimiteri nisu svugde isti, vec delimitera nema tamo gde se zna tacna duzina polja, a dva space-a su tamo gde je ispred polje koje moze da sadrzi space, plus na nekim mestima je delimiter samo jedan space (verovatno izmedju cifara).

Na ovako malom uzorku koji si dao, ne moze sa sigurnoscu sve da se utvrdi, ali neka zakonitost postoji.

16.02.2007. u 14:16 

mVeliki
Miroslav Ćurčić
Novi Sad

Član broj: 19034
Poruke: 568
*.dynamic.sbb.co.yu.



Profil

icon Re: Import iz velikog .txt-a u mysql - više pitanja16.02.2007. u 14:25
Pitaj šefa ko mu je dao taj fajl, nek napravi upotrebljiv export,
recimo tab-separated ili zero-separated.

Sigurno nije taj fajl korišten takav kakav je.
"The quieter you become, the more you are able to hear."
16.02.2007. u 14:25 

bslijepcevic
Branko Slijepcevic
Nis

Član broj: 114634
Poruke: 11
*.dynamic.sbb.co.yu.



Profil

icon Re: Import iz velikog .txt-a u mysql - više pitanja20.02.2007. u 10:09
Ne bi bilo lose da probas da pronadjes neki opensource CSV importer.Imas ih tonu uradjenih u PHP-u i da probas.Samo bi bilo pozeljno da prebacis file na server pre nego sto pocnes importovanje, a sto se velicine fajla tice neces imati problema jer CSV parsuje liniju po liniju i izvrsava INSERT tako da velicina fajla nije bitna.
Nadam se da ce ovo resiti tvoj problem
20.02.2007. u 10:09 

sale83
Aleksandar Ervacanin
Full Time Student & Full Time Zaposlen ( IT Suppo..
Sydney

Član broj: 41625
Poruke: 607
*.ispone.net.au.

Sajt: www.serbiansite.com


Profil

icon Re: Import iz velikog .txt-a u mysql - više pitanja21.02.2007. u 05:52
Pa sto se tice rastavljanja tog primera sto si dao to i nije nesto tesko ....

Jeste da je sve traljavo uradjeno ali da se srediti........


U sustini ja bi to ovako uradio ako vec moram da parsujem taj string da bi dobio ono sto trazis...


1)Razbiji string na onaj veliki SPACE( )
tj ova dva u ovom delu:
Code:

... usluge     PONTIN D.O.O.     0600888913...

Tako dobijas tri nova stringa.

2) Prvi string od gore tri dobijena moras da rastavis na " " i za to koristi strpos() substr() tako dobijas onaj ID i text.

3) Drugi string nediras jer on je OK i taj treba da ide u bazu.

4) E treci string je sad malo veci problem...
On treba da sadrzi ovo:
Code:

0600888913 1999070613Zadarska  5207 Zadar  71951 Zadar  23000Gazenicka Cesta  32 5190019950217 999

To mozes da razbijes sa preg_split("/[\s,]+/", Treci_String_ovde);

5)Sad kad si razbio i taj treci string dobijas novi niz .. Problem kod novog niza je clan niz[1] i niz[6] zato su su brojevi zajedno sa stringom a tebi trebaju odvojeni....

Mada i to nije neki problem....

Iskoristis for($=0...) { ...} Prodjes kroz niz i ako je $i =i ili $i =6 onda te clanove niza razbijes opet...
recimo ovako nekako

Code:

preg_match('#[\d]+#', $NoviNiz[$i], $matches); // Gde  je $i = 1 ili 6
//$matches[0]; // Brojevi koji ti trebaju
preg_match('#[A-Za-z]+#', $polja_1[$i], $matches1); // Gde  je $i = 1 ili 6
//$matches1[0]; // Slova ....



6) E sad kad si sve razbio morao bi da vratis niz svega toga.... ...

7) Kao rezultat toga niza dobio bi ovako nesto;

// PS nadam se da je ovako. Ovo napamet ti pisem..... Tj mislim da taj scenario treba da vrati ovako nesto
Code:

0 => 1000039 
1 => PONTIN drustvo s ogranicenom odgovornoscu za ribarstvo,preradu ribe, trgovinu, turizam i usluge
2 => PONTIN D.O.O.
3 => 0600888913
4 => 1999070613
5 => Zadarska
6 => 5207
7 => Zadar
8 => 71951
9 => Zadar
10 => 23000
11 => Gazenicka
12 => Cesta
13 => 32
14 => 5190019950217
15 => 999



8) E sad vidis posto tebi treba adresa onda je to valjda od ovog gore niza sto dobijes clan [11][12][13] :)

Sad treba taj niz da propustis kroz for ili foreach sta god ti oces i lepo kreiras novi niz s time da clanove [11][12][13] treba da implodujes i sastavis kao jedan clan jer to je Adresa...

Problem nastaje kod toga da ADRESA nije uvek ista... E ona ces morati da proveravas 14 => 5190019950217 tj da kroz do while impoldujes sve clanove niz od 11 skroz dok neki clan niza nije jedank necem slicnom kao 5190019950217 ( za ovaj broj moze da koristis Regular expression, is_numeric + strlen() ) ....


Jbg stao mi mozak ne mogu vise .... Valjda ce ti ovo biti od pomoci ako su svi redovi taki kao sto je ovaj primer..

Ako nisu onda Srecno....


PS Kao sto gore rekose:

pozdravi majstora koji je formatirao csv


Poz
sale



PHP/MySQl/SQL/VB.NET/JAVA/JSP/Servlet/MSSQLServer/MSAccess/Perl
COBOL/JCL/CICS -> IBM Mainframe
Nortel/Cisco/SideWinder
21.02.2007. u 05:52 

asyk

Član broj: 46486
Poruke: 55
*.adsl.net.t-com.hr.



Profil

icon Re: Import iz velikog .txt-a u mysql - više pitanja22.02.2007. u 08:22
Prvo, hvala svima koji pokušavaju pomoći, to se cijeni!

Dakle, nakon mucenja i mucenja, uspio sam doci do nekakvog ajmo reci malo normalnije formatiranog dumpa te baze. Ovako podaci izgledaju sad:

Code:

1000039 PONTIN društvo s ograničenom odgovornošću za ribarstvo,preradu ribe, trgovinu, turizam i usluge                                                                                                                                                                PONTIN D.O.O.                                                                                                                                   0600888913 1999070613Zadarska                                          5207 Zadar                                             71951 Zadar                                             23000Gaženička Cesta                                   32      5190019950217                    999
1000101 B I R O M A  D.O.O. ZA TRGOVINU UREDSKOM OPREMOM I KANCELARIJSKIM MATERIJALOM                                                                                                                                                                                  B I R O M A  D.O.O.                                                                                                                             080295620  1999052722Grad Zagreb                                       1333 Grad Zagreb                                       72150 Zagreb                                            10000ČIKOŠEVA                                            8     5248119950213                    999
1000217 OPORTUN d.o.o. za informatiku i trgovinu                                                                                                                                                                                                                       OPORTUN d.o.o.                                                                                                                                  0700381833 199902185 Varaždinska                                       2895 Novi Marof                                        46442 Paka                                              42220Paka                                               16     7222019950213                    999
1000241 PERFETA D.O.O. ZA UNUTARNJU I VANJSKU TRGOVINU, ZASTUPANJE I POSREDOVANJE                                                                                                                                                                                      PERFETA D.O.O.                                                                                                                                  040088309  199709038 Primorsko-goranska                                3735 Rijeka                                            55174 Rijeka                                            51000KAMPANJA                                           21     5190019950217                    999
1000250 JERKUNICA, društvo s ograničenom odgovornošću za računovodstvene poslove, trgovinu i turizam                                                                                                                                                                   JERKUNICA d.o.o.                                                                                                                                0600905491 1998010917Splitsko-dalmatinska                              4090 Split                                             59200 Split                                             21000Table                                              10     7412019950217318673    318673    999
1000268 UNISOFT D.O.O. ZA INFORMATIČKI INŽENJERING                                                                                                                                                                                                                     UNISOFT D.O.O.                                                                                                                                  040094970  1997100718Istarska                                          4685 Umag                                              66761 Umag                                              52470VLADIMIRA NAZORA                                    5     7222019950217                    999
1000420 PREMIK društvo s ograničenom odgovornošću za trgovinu, graditeljstvo i ugostiteljstvo                                                                                                                                                                          PREMIK d.o.o.                                                                                                                                   0801854972 200007031 Zagrebačka                                        345  Brdovec                                           25976 Javorje                                           10291Nova ulica                                        19      5190019950213                    999
1000438 TERNO društvo s ograničenom odgovornošću za građevinarstvo i usluge                                                                                                                                                                                            TERNO d. o. o.                                                                                                                                  08128328 2 199804171 Zagrebačka                                        3620 Pušća                                             15512 Dubrava Pušćanska                                 10294Dubrovačka Cesta                                  49      4521119950213                    999
1000454 JAGER D.O.O. ZA TRGOVINU I UGOSTITELJSTVO                                                                                                                                                                                                                      JAGER D.O.O.                                                                                                                                    040104006  199803028 Primorsko-goranska                                3735 Rijeka                                            55174 Rijeka                                            51000RADIĆEVA                                           14     5190019950213                    999
1000489 MADENA društvo s ograničenom odgovornošću za trgovinu i usluge                                                                                                                                                                                                 MADENA d. o. o.                                                                                                                                 0400203444 200010048 Primorsko-goranska                                3735 Rijeka                                            55174 Rijeka                                            51000Janka Polića Kamova                               101     5190019950213450099    450200    999
1000560 GO - MO, D.O.O. ZA UGOSTITELJSTVO,TURIZAM I TRGOVINU                                                                                                                                                                                                           GO - MO, D.O.O.                                                                                                                                 040043892  1997011618Istarska                                          4685 Umag                                              66761 Umag                                              52470MATTEA BENUSSIA                                     1     5540019950216                    999
1000586 LITUS, društvo s ograničenom odgovornošću za proizvodnju, trgovinu i usluge                                                                                                                                                                                    LITUS, d.o.o.                                                                                                                                   0601255692 1999060518Istarska                                          3484 Poreč                                             50849 Poreč                                             52440Pulska                                            9       5190019950213434603    434603    999
1000632 NEAPOLIS, društvo s ograničenom odgovornošću za projektiranje, gradnju, upravljanje, prodaju i iznajmljivanje objekata, turizam, trgovina i export-import                                                                                                      NEAPOLIS  d.o.o.                                                                                                                                0400915408 2004021618Istarska                                          2917 Novigrad                                          43982 Novigrad                                          52466Ulica Gradska Vrata                                29     7420019950216                    999
1000675 PRAGMA društvo s ograničenom odgovornošću za poslovne usluge                                                                                                                                                                                                   PRAGMA d.o.o.                                                                                                                                   04006197 3 200003208 Primorsko-goranska                                3735 Rijeka                                            55174 Rijeka                                            51000Bačvarska                                         1       4532019950214                    999
1000713 VIZUAL d.o.o. za trgovinu i usluge                                                                                                                                                                                                                             VIZUAL d.o.o.                                                                                                                                   0300181282 2004022014Osječko-baranjska                                 3123 Osijek                                            45691 Osijek                                            31000Stjepana Radića                                   32      5190019950213                    999
1000756 ŽE-PA, D.O.O. ZA PROIZVODNJU, TRGOVINU, UGOSTITELJSTVO I USLUGE                                                                                                                                                                                                ŽE-PA, D.O.O.                                                                                                                                   020019744  199611289 Ličko-senjska                                     3131 Otočac                                            46094 Otočac                                            53220KRALJA ZVONIMIRA                                   95     6024019950215                    999
1000853 COMMERCE KLJAJIĆ D.O.O.ZA IZGRADNJU, TRGOVINU, USLUGE I UVOZ-IZVOZ                                                                                                                                                                                             COMMERCE KLJAJIĆ D.O.O.                                                                                                                         050018602  1996050912Brodsko-posavska                                  108  Bebrina                                           72982 Zbjeg                                             35254                                                      bb  4521119950216                    999
1000870 URKA d.o.o. za trgovinu, turizam i usluge.                                                                                                                                                                                                                     URKA d.o.o.                                                                                                                                     0600270103 2005060719Dubrovačko-neretvanska                            981  Dubrovnik                                         15709 Dubrovnik                                         20000Andrije Hebranga                                   33     5190019950214                    999
1000993 EGMOND proizvodnja i promet robom, d.o.o.                                                                                                                                                                                                                      EGMOND d.o.o.                                                                                                                                   0400816193 199910298 Primorsko-goranska                                698  Delnice                                           10634 Delnice                                           51300Kamenita                                          7       3110019950213812520    812520    999
1001019 MAKROSISTEM - INŽINJERING društvo s ograničenom odgovornošću za proizvodnju, unutrašnju i vanjsku trgovinu i usluge                                                                                                                                            MAKROSISTEM - INŽINJERING d.o.o.                                                                                                                0802794961 1999031122Grad Zagreb                                       1333 Grad Zagreb                                       72150 Zagreb                                            10000Male Putine broj 2                                        7222019950213                    999
1001027 MARLES KUĆE  građevinarstvo, trgovina i usluge d.o.o.                                                                                                                                                                                                          MARLES KUĆE d.o.o.                                                                                                                              0600335904 2001010822Grad Zagreb                                       1333 Grad Zagreb                                       72150 Zagreb                                            10000Zrinjevac                                         16      4521119950213