Majestic-12

[Norman]

nächste und wahrscheinlich letzte beta 4 ist draußen und arbeitet gut
ist i.m. 50% schneller und da gzip bei crawlen verwendet wird wesentlich sparsamer bei der nutzung der bandbreite.
dank dem neuen feature um die uploadpakete zu encoden erlaubt es auch usern die von ihrem ISP etwas geblockt werden nun ohne störungen zu uploaden

http://www.majestic12.co.uk/forum/viewtopic.php?t=2967

Ok, BETA 4 - I hope the last one Shocked

This is not terribly different from BETA 3.2, though if you run it then please upgrade - this I hope will be final beta and then we go for release, I think we need to run it at least maybe 4-5 days, or even whole of next week to be sure we have got stable replacement for fairly well working v1.5.0.

Unfortunately I had to disable disk IO saving tech in Mono builds due to issues - I really overrun on time that I planned for new node so I just don't have energy to finish this bit - all the other benefits will still work on Mono - mainly gzip support and archiving should still be a considerably faster. I will revisit this next month - I am sure solution is fairly easy I am just out of energy and we have to push very hard on anchor index improvements as our future in many respects depends on success there.

Windows

.NET 2.0 build (recommended): http://majestic12.co.uk/files/mj12node/ ... v160b4.zip

.NET 1.1 build: http://majestic12.co.uk/files/mj12node/ ... v160b4.zip

------------------------------

Note on Mono builds: if you used BETA 1 and had .GZ files created, then those will be deleted at start and new format .GZ files will be created - you can see some text messages about it, those buckets will be restarted.

Linux: http://majestic12.co.uk/files/mj12node/ ... v160b4.tgz

FreeBSD: http://majestic12.co.uk/files/mj12node/ ... v160b4.tgz

Mac: http://majestic12.co.uk/files/mj12node/ ... v160b4.tgz

[Norman]

wieder ne neue beta ( 1.6.0 beta 6 )

[achims311]

wieder ne neue beta ( 1.6.0 beta 6 )

Gibts auch release notes? (Und welche version windows oder linux?

[Norman]

This just does not work well for me today, does it? Crying or Very sad

Sorry guys, found another bug in BETA 5.1 and had to force through this BETA 6: it's not my day today for sure, or maybe whole week if not longer Crying or Very sad

On a plus side this build implements "sync points" for barrels - this should prevent incorrect "barrel corrupted" issues as if one page is deemed broken then only that page will be skipped, not rest of barrel.

Windows

.NET 2.0 build (recommended): http://majestic12.co.uk/files/mj12node/ ... v160b6.zip

.NET 1.1 build: http://majestic12.co.uk/files/mj12node/ ... v160b6.zip

Linux: http://majestic12.co.uk/files/mj12node/ ... v160b6.tgz

FreeBSD: http://majestic12.co.uk/files/mj12node/ ... v160b6.tgz

Mac: http://majestic12.co.uk/files/mj12node/ ... v160b6.tgz

Wird höchstwahrscheinlich als release rausgehen.
läuft auch sehr gut wie ich finde.

Code: Alles auswählen

MJ12node       :	v1.6.0 BETA 6 (.NET 2.0)
Platform       :	Win32 specific running on Microsoft Windows NT 5.1.2600 Service Pack 2
Total URLs     :	4.719.182 (100,0%)
Successes      :	3.927.049 (83,2%)
Not found      :	194.674 (4,1%)
Timed out      :	79.658 (1,7%)
Disallowed     :	70.110 (1,5%)
Banned         :	17.724 (0,4%)
DNS errors     :	264.074 (5,6%)
Conn errors    :	41.940 (0,9%)
Forbidden (403):	7.464 (0,2%)
Other          :	117.009 (2,5%)
Retries        :	241 (5,0%)
Uptime         :	1 day 10 hours 54 mins 20 secs
Memory usage   :	134 MB
GZIP requests  :	814.089 (17,3% of successes)
GZIP saved data:	24.750 MB (18,9% of total)

[Norman]

RC1 erschienen:

Ok, we are almost at the end of the road for new release - RC stands for Release Candidate and I hope we won't have more - this is effectively BETA 6 with a couple of bugs fixed for found bugs yesterday, I don't really want to touch node with a bargepole for at least a few months - the main question to answer right now is whether this new version is ready for release for everyone to upgrade to it.

Let's run this for a few more days and then come to a conclusion if it can be finally released - if anyone was staying away from beta testing, then it is a good time right now to check it out as we almost ready to release it. Smile

Windows

.NET 2.0 build (recommended): http://majestic12.co.uk/files/mj12node/ ... 160rc1.zip

.NET 1.1 build: http://majestic12.co.uk/files/mj12node/ ... 160rc1.zip

Linux: http://majestic12.co.uk/files/mj12node/ ... 160rc1.tgz

FreeBSD: http://majestic12.co.uk/files/mj12node/ ... 160rc1.tgz

Mac: http://majestic12.co.uk/files/mj12node/ ... 160rc1.tgz

[Norman]

RC2 ist da:

This is it - the final RC2 build - I thought we could get away with RC1 which seemed very good, but I decided to add more robust handling of archiving errors: it seems to happen very rarely now, but still, if some barrel for some reason will fail to archive now more than 3 times in a row, then it won't stall node by trying to archive same thing and fail again and again, it will be set aside and other buckets will be archived. This way in case of weird error node won't be completely disabled by one bad barrel that can't be archived (for whatever reason).

Also I added shortcuts (key combinations) to enable Snooze Profiles - you can see them in Windows builds in Profiles menu, this was asked for by hixie.

Also bear in mind we are going through last buckets right now so error rates could be higher - don't jump to conclusions insofar those are concerned, RC2 is the same crawling wise as RC1, we should get new quality buckets in 48 hours or so.

This is the final test build of v1.6.0 - I think we are ready to release it, but since I made some changes I Want to be sure I did not screw anything up, if all goes well next week (as I expect) then we get new node released for everyone Cool

If anyone was sitting on the fence deciding not to use new node due to dislike of beta testing then I think it is good time you give it a go - it seems pretty stable and seems all key initial bugs found by beta testers now fixed Smile

Windows

.NET 2.0 build (recommended): http://majestic12.co.uk/files/mj12node/ ... 160rc2.zip

.NET 1.1 build: http://majestic12.co.uk/files/mj12node/ ... 160rc2.zip

Linux: http://majestic12.co.uk/files/mj12node/ ... 160rc2.tgz

FreeBSD: http://majestic12.co.uk/files/mj12node/ ... 160rc2.tgz

Mac: http://majestic12.co.uk/files/mj12node/ ... 160rc2.tgz

[Norman]

nun ist das release raus:

http://www.majestic12.co.uk/forum/viewtopic.php?t=3010

Important release with the big changes (get it from download page):

Code:

v1.6.0 10/04/08
+ 4-5 times reduction in temporary space usage as well as Disk IO
+ Much faster and more error proof archiving
+ Support for GZIP during crawling
! Lots of bug fixes.


It is strongly recommended to upgrade to this release - much faster crawling (25-30+% more) for the same bandwidth, plus a lot less disk space used!

[Velociraptor]

nun ist das release raus...

wuhuu gleich mal testen ^^

[Norman]

läuft und läuft und läuft und läuft und läuft und läuft und ....

[Norman]

neue version released:

This build features better support for "deep-crawling" of a small number of important domains (like blogspot.com) that contain a lot of urls on them.

From history.txt:

Code:

v1.6.2 11/07/08
! Fixed bug that with some fairly rare crawled data case
would make archiver use a lot more memory and CPU than expected
! Fixed not counting some pages as successfully downloaded in grid on Activity screen
+ Added options to control number of deep crawl buckets as well as ability to prioritise them
! Default .NET usage of proxy if it was specified in Internet Explorer is now disabled
in .NET 2.0 build: MJ12node.exe.config - <defaultProxy enabled="false" useDefaultCredentials="false" />
Proxy (if used) will need to be specified explicitly in Option->Connection.


Download new node from here.: upgrading is simple - shut down current node and unzip archive into nodes directory overriding all files - this will NOT override config/crawling progress. You can always roll back to previous stable version v1.6.1.

[Jkcapi]

Bin auch wieder dabei!
Wenn jetzt auch noch Firstomega wieder mitmachen würde, dann könnten wir echt was reißen

Gruß J.Kampmeier

[Roland Schneider]

das war die tagesstatistik, aber da du dich schon auf 16 hochgearbeitet hast ist der 6. in der gesamtstatistik auch nicht weit

weiter so, und schön, dass mal ein paar leute mehr mei m12 gas geben

und wenn ich mir firstomega anschaue sagt mir dass, dass wohl da irgendwie ein leitungsupgrade auf 16MBit fällig war

Nein, waren sogar 26 MBit

Bin auch wieder dabei!
Wenn jetzt auch noch Firstomega wieder mitmachen würde, dann könnten wir echt was reißen

Gruß J.Kampmeier

Leider bin ich inzwischen wieder umgezogen und am neuen Wohnort gibt's kein Internet über Kabel