Joined: Thu Jan 11, 2007 21:34
Posts: 77

Few problems after heavily using ntfs-3g

I used NTFS-3g for e.g. filling an 800 GB partition with data for a few months now and ancountered following problems:
1. Sometimes, NTFS-3g seems to allocate extern space for the security descriptors instead of saving them in the MFT (a changing of the security setting for all files in Vista solved that)
2. Even then, it seems that the place in which the SDs was saved belongs to 'System' according both Diskkeeper and O&O defrag, which makes defragment nearly impossible.
3. The 'Analysing' both with chkdsk, Diskeeper and O&O is much slower if the files are saved with NTFS-3g than with Vista, I hear much more seeks on the HDD, too. That has nothing to do with MFT and/or data fragmentation, MFT is in 2-3 pieces, only 500 files from 400k are fragmented, but analysing is only at a speed about 200 files/second.
4. It can depend on the other problems, but while the allocator of NTFS-3g seems to be a lot better than Vistas in view of fragmentation, it seems that it likes to spreed the files over the whole HDD, which is very bad if I linearly copy 700 GB fairly big files (mostly 5 MB-20 GB) to a 800 GB partition. It was extreme on a 50 GB partition filled with 20 GB little files (average ca. 20-50 kb) - it was like a chessboard on the blockview on O&O^^

Btw, I'm using Ubuntu 8.10 and 9.04, so I should have had always fairly current versions (e.g. now I've 2009.2.1).

Joined: Tue Nov 21, 2006 23:15
Posts: 1648

Re: Few problems after heavily using ntfs-3g

Hi,
1. What do you mean by 'extern space'? What security setting did you change in Vista?
At the moment Security Descriptors always need extra space in the MFT record. The Advanced NTFS-3G driver solves this because it uses the Security Indexes. I think it would be very nice if the Stable NTFS-3G driver would do the same as the next step for merging.
2. Why do you think they are Security Descriptors and not something else? Defrag is certainly not impossible, only Windows doesn't support it.
3. How much is the 'much more'? All of them use the Microsoft NTFS driver and if it's indeed very inefficient then we could release a fast and efficient NTFS defragmenter.
4. The strategy is very good for copying large files. But not so much for many small files. At the moment this how we minimize file fragmentation. Extreme file fragmentation would significantly slow down things.
Regards, Szaka

1.1 External space - 'normal' space, not in the MFT - that can be prooved because when I changed the SDs I got a few MB freed.
1.2 security settings - I meant the permission rights
2. Because on these areas there where the SDs (seen that wifh O&O defrag) before I changed all security rights which I saw at locked files on O&O - I know that there can be mft-records outside the 'main' $MFT-file, right? Perhaps Vista deleted only the SDs in this 'extra mft spaces' (don't know how there are called officially), but not the 'extra mft spaces' itself?
3. much more - Analysing files wich chkdsk/O&O created with Vista are at speed at ca. 5000 files/sec, created with NTFS-3f at only 100...200 files/sec
Btw, I'm on freenode, my name there is mifritscher, too, if you have further questions.

Joined: Tue Nov 21, 2006 23:15
Posts: 1648

Re: Few problems after heavily using ntfs-3g

1.1 The NTFS-3G created SD is 80 bytes. This easily fits into the 1024 bytes large MFT record. There is a very-very minimal chance that NTFS-3G ever pushes out an SD from the MFT record.
What I think is that Vista does this. It can be confirmed if you send the ntfsclone metadata image to szaka@ntfs-3g.org following the instructions at http://www.linux-ntfs.org/doku.php?id=n ... s_metadata and tell me which files are the 'problematic'.
2. There are no MFT records outside the $MFT file.
3. How do you run chkdsk so you can see such stats as 5000 files/s, 200 files/s?

chkdsk /f x:
under Vista, it shows e.g. x/x files are worked or so, I calced this speeds by viewing them, so there aren't that accourate^^
Ok, I'll send you the metadata soon - but I already changed all permissions :-/

hmm:
michi@michis-ibm:/tmp$ sudo ntfsclone -f --ignore-fs-check --metadata --output metadata.img /dev/mapper/truecrypt2
ntfsclone v2.0.0 (libntfs 10:0:0)
NTFS volume version: 3.1
Cluster size : 4096 bytes
Current volume size: 53686956032 bytes (53687 MB)
Current device size: 53686960128 bytes (53687 MB)
Scanning volume ...
ERROR: Cluster 1638402 referenced twice!
You didn't shutdown your Windowsproperly?
Strange, because I actually let shutdown Vista probably and run a chkdsk this day...
Also it is strage that the option ignore-fs-check is ignored...
Trying another volume (this time the 850 GB one)

Joined: Tue Nov 21, 2006 23:15
Posts: 1648

Re: Few problems after heavily using ntfs-3g

The 'Cluster 1638402 referenced twice!' is a fairly bad error message. It means two files use the same data block which must never happen.
I really don't know how much ntfsprogs 2.0.0 can be trusted. It definitely has many problems though I'm not aware ntfsclone would have this kind of bug.
You may try ntfsprogs 1.13.1 and if the error is the same then chkdsk should fix the problem.
But first please run 'ntfscluster -fc 1638402 /dev/mapper/truecrypt2 2> /dev/null'. It should tell which files overlap.

ok, ntfscluster does only find one file...
I'm looking for a 1.13.1 version of ntfsprogs now.

Interesting: oth 1.13 and 2.0 ntfscluster show only one file, but both ntfsclone have the same behavior...

Joined: Tue Nov 21, 2006 23:15
Posts: 1648

Re: Few problems after heavily using ntfs-3g

Please try this one http://ntfs-3g.org/download/ntfs-3g.clone-1.5388.tgz
then you can do
ntfs-3g.clone -mso - DEVICE | bzip2 > DEVICE-meta.img.bz2
This is much faster than ntfsclone.

ok, this version worked for the 50 GB volume I've sent you the link.
the 'big' volume is still beeing worked on ;)

now, also the big volume is sent.

Joined: Tue Nov 21, 2006 23:15
Posts: 1648

Re: Few problems after heavily using ntfs-3g

Two files (017.part.met and 027.part) indeed overlap:

Both are related to eMule and one of them was updated by Windows the other by Linux. It's impossible to tell which one made the error but we do know that Vista has reliability problems with sparse files. Afair, eMule (or some torrent client?) recommends using preallocated files because of this. I don't know how this can be configured.
I'm also sorry to say that apparently you have 12 corrupt file/directory. Hopefully this is only an nfts-3g.clone bug which is still in development. Please do a 'find >/dev/null' at the root directory then check the log files for error. I can not access four directories at the top directory.

running :-)
Ah - I remember now - I got 2 BSODs shorttime after starting amule (page fault in nonpages area)
I runned emule first under Vista, then under Linux. Hmpf, I didn't linked the crashs with NTFS-problems :-/
Additionally, a find over the truecypt2-volume showed no error...

Hmm, and the truecrypt4 volume was defragmented by PerfectDisk - but was remounted after the first crash, but not used, so I think this volume should be ok beause of the journaling.
Perhaps ntfs-3g shouldn't simply omit the journaling-data?
The next thing I do is going to Vista again and do a full chkdsk.

following errors were reported on truecrypt2:
CHKDSK überprüft Dateien (Phase 1 von 3)...
Beschädigter Attributeintrag (128, ') wird Datensätzen verarbeitet)
vom Datensatzsegment 67050 gelöscht.
75 Prozent abgeschlossen. (252192 von 259472 Indexeinträgen verarbeitet)
Indexeintrag 017.part.met in Index $I30 der Datei 66979 wird gelöscht.
75 Prozent abgeschlossen. (252245 von 259472 Indexeinträgen verarbeitet)
259472 Indexeinträge verarbeitet.
Indexüberprüfung beendet.
CHKDSK stellt verlorene Dateien wieder her.
1 nicht indizierte Dateien verarbeitet.
Verwaiste Datei 017.part.met (7845) wird in Verzeichnisdatei 66979 wiederhergest
ellt.
Verwaiste Datei 017PAR~1.MET (7845) wird in Verzeichnisdatei 66979 wiederhergest
ellt.
99 Prozent abgeschlossen. (250429 von 251456 Beschreibungen verarbeitet)
99 Prozent abgeschlossen. (251099 von 251456 Beschreibungen verarbeitet)
251456 Sicherheitsbeschreibungen verarbeitet.
2 unbenutzte Indexeinträge werden aus dem Index $SII der Datei 9 gelöscht.
2 unbenutzte Indexeinträge werden aus dem Index $SDH der Datei 9 gelöscht.
2 nicht verwendete Sicherheitsbeschreibungen werden aufgeräumt.
Spiegelkopie des Datenstroms der Sicherheitsbeschreibungen wird berichtigt.
Überprüfung der Sicherheitsbeschreibungen beendet.
Attribut DATA wird in Datei 67050 eingefügt.
4009 Datendateien verarbeitet.
Fehler im Attribut BITMAP der Masterdateitabelle (MFT) werden berichtigt.
CHKDSK hat freien Speicher gefunden, der in der Volumebitmap als
zugeordnet gekennzeichnet ist.
Windows hat Probleme im Dateisystem behoben.
So only a few errors :-))

truecrypt4 was ok (despite of the extreme slowness of chkdsk)

Joined: Tue Nov 21, 2006 23:15
Posts: 1648

Re: Few problems after heavily using ntfs-3g

> CHKDSK überprüft Dateien (Phase 1 von 3)...
> Beschädigter Attributeintrag (128, ') wird Datensätzen verarbeitet)
> vom Datensatzsegment 67050 gelöscht.
This is the 027.part file I noted above and which overlaps with the below
017.part.met file.
There is absoluely nothing wrong with the NTFS metadata of the file.
Chkdsk removed it because one of its clusters (blocks) was already
referenced by the 017.part.met file.
> Indexeintrag 017.part.met in Index $I30 der Datei 66979 wird gelöscht.
This is the other file I noted above which overlaps with 027.part. It has
another problem. The directory index was created by Linux and it was
correct but Windows (defrag?) moved the file into a different MFT record
but it didn't update the index. It still points into the old, valid MFT
record which got marked unused.
This seems to be either another Windows NTFS driver bug or you had a system
crash during defragmentation.
> CHKDSK stellt verlorene Dateien wieder her.
> 1 nicht indizierte Dateien verarbeitet.
Refers to 017.part.met.
> Verwaiste Datei 017.part.met (7845) wird in Verzeichnisdatei 66979
> wiederhergest ellt. Verwaiste Datei 017PAR~1.MET (7845) wird in
> Verzeichnisdatei 66979 wiederhergest ellt.
Windows finishes moving the file and updates the directory index which
didn't happen earlier.
> 2 unbenutzte Indexeinträge werden aus dem Index $SII der Datei 9
> gelöscht. 2 unbenutzte Indexeinträge werden aus dem Index $SDH der
> Datei 9 gelöscht. 2 nicht verwendete Sicherheitsbeschreibungen werden
> aufgeräumt.
http://ntfs-3g.org/support.html#cleaningup
> Attribut DATA wird in Datei 67050 eingefügt.
Here CHKDSK replaced a good file with an empty one.
> Fehler im Attribut BITMAP der Masterdateitabelle (MFT) werden berichtigt.
> CHKDSK hat freien Speicher gefunden, der in der Volumebitmap als
> zugeordnet gekennzeichnet ist.
http://ntfs-3g.org/support.html#usedfreespace
Overall the only problem was the one what ntfsclone also found.

yes, this problem wasn't so harmfull and almost expected after I realized that the Vista-crashs had something todo with NTFS.
The main questions for me are: Why were the SDs saved externally, and why are this places still shown as 'locked'?

Joined: Tue Nov 21, 2006 23:15
Posts: 1648

Re: Few problems after heavily using ntfs-3g

I couldn't find yet any external SD. So far all of them were in the base MFT record.

yes, there all moved to the MFT by changing the permissions.
I'll try to create a image with external SDs.
Hmm, I experimented with the 'advanced' version of the ntfs-3g driver for a few weeks, could it be that this driver created them?

Joined: Tue Nov 21, 2006 23:15
Posts: 1648

Re: Few problems after heavily using ntfs-3g

SDs are always in the $MFT. Maybe you mean non-base MFT records, i.e. extended MFT record.
Truecrypt2 has 485 SDs, all in base MFT records. Truecrypt4 has 11 SDs, all in base MFT records.

It shouldn't do that but maybe that has something to do with the issue you experience.

Joined: Tue Sep 04, 2007 17:22
Posts: 1286

Re: Few problems after heavily using ntfs-3g

Hi,

The Advanced ntfs-3g store the SD (security descriptors) like Windows does : they are grouped in the special file $Secure and a reference is put in the base record of the file, while the standard ntfs-3g creates a special entry, generally in the base record of the file. Some defragmenters have already been mentioned as disturbed by these special entries (see viewtopic.php?f=2&t=864), but you should not experience that with Advanced ntfs-3g (unless some unknown problem occurred).
With Advanced ntfs-3g, like with Windows, a single entry is created in $Secure for all the files which have the same owner and protection, but when an entry is not needed any more (because the protections of all the files which were using it have been changed), the entry is not freed and can be used for a subsequent file using the said owner and protection. This is why you get the following warnings :

$Secure is file 9 and $SII and $SDH are two indexes for fast access to a SD, and you get two unused entries, which is quite normal. chkdsk has reclaimed their space, and they will be recreated if they are needed again.

Due to the grouping of the SD, such a big count is very unlikely created by Advanced ntfs-3g or Windows. For a single user, using all the values for chmod from 0000 to 07777, at most 4096 SD can be created in $Secure, and 251456 means at least 62 user+group pairs.
What exactly is the remaining problem with the SD ?
Regards
Jean-Pierre

Joined: Tue Sep 04, 2007 17:22
Posts: 1286

Re: Few problems after heavily using ntfs-3g

Hi,
It just occurred to me that currently the cache size for the SD is limited to 262144 entries in $Secure. This is not the limit on the number of SDs, if there are more SD, they just cannot be cached.
Is it possible you really have 251456 entries in $Secure, not counting the entries created by standard ntfs-3g (which cannot be cached). This would mean a lot of users... or data corruption.
Regards
Jean-Pierre

Joined: Sun Feb 14, 2010 11:42
Posts: 1

Can't mount my NTFS disk.

I buy the software, restart, and I get this error.
Help?

Hi, I got the same problems.
I have got a 1 TB external hard drive with NTFS-format, and one with 500 MB
I loaded the following ones, and tryed both in relation to macfuse to be able writing an reading the hard drives with my Mac:
--> tuxerantfs_2010-1.dmg
--> ntfs-3g-2010.1.16-macosx.dmg
What happened was, that my Mac was unable to mount the external hard drive and showed those error messages i loaded up.
I did as I was told in the messages and plugged in the disk at my windows-PC and tried a 'chkdsk'.
The result was, that my Mac again mounted the hard drive and I was able to read files.
Idecidet to uninstall tuxerantfs and ntfs-3g but it didnt work.
I tryed to uninstall them via the Systemmanager and the Terminal too and it seemed as if it worked... BUT it didnt!
I still got errormessages with my secon hard drive which I didnt plugged in to my windows-PC to make a chkdsk - although I uninstalled the programm - macfuse too by the way.
After about 4 hours of big fun I thought to myself: 'Okay, fat32 will handle it!' and Itried to.
The texurantfs kind of 'killed' several functions of the mac Disk-Utility-tool.
There wasnt any longer the format 'fat32' which was there before I installedtexurantfs and ntfs-3g.
This function and others got los. In adition there were other new formats to choose like 'Windows NT Data System (NTFS-3G)' and 'Windows NT Data System (TUxera NTFS)'
>
How can I delete ntfs-3g and TuxeraNTFS completly from my Mac
AND
what do I have to do for repairing my Disk.Utility-Tool?
Please Answer, its just a ***** situation

Isnt ther anybody whoknows a solution or got some advices? *Help*

Hi, I've got the exact same problem and error message as Lefu. Has anyone been able to solve this problem?
I've tried running the chkdsk /f on windows, but it gives a message saying that it's in NTFS format and it's not able to run it because other processs are using the disk. I cannot mount it on my Mac still, presumably because I haven't successfully run the chkdsk. Any help here?

I doesn't look like this software really has any support. Plenty of problems posted here, but not many solutions. I do not see much help for anybody in this forum. I posted a problem up yesterday but it still has yet to be 'approved' by the moderators for posting.

Joined: Sat Oct 06, 2007 09:24
Posts: 530

Re: Can't mount my NTFS disk.

Hi,

Support for Tuxera NTFS for Mac customers is handled through a dedicated support email address, and not normally through the forum.
Regarding NTFS-3G, its free and open source, and we'll try to answer questions related to NTFS-3G when we have time.
I think we have managed to answer most of the recently posted questions.
Regards,
- Erik

Joined: Sat Oct 06, 2007 09:24
Posts: 530

Re: Can't mount my NTFS disk.

Did you contact the dedicated support email address described in your activation email? This is your priority support email, and the preferred way for our customers to get help.
Please see the following thread: http://www.tuxera.com/forum/viewtopic.php?f=4&t=5021
Regards,
- Erik

Joined: Sat Oct 06, 2007 09:24
Posts: 530

Re: Can't mount my NTFS disk.

Right... there were some minor inconsistencies on the drive. NTFS-3G and Tuxera NTFS for Mac are both very careful about not doing anything that could possibly endanger your data. That is why it requires you to repair your on-disk structures if they seem inconsistent before using your drive with our software.

How did you attempt the uninstall, and what were the error messages when uninstalling?

What commands did you use for uninstalling, and what error messages did they result in?

NTFS-3G / Tuxera NTFS for Mac certainly does not touch any files related to other filesystem types, so that sounds virtually impossible. Could you give me a list of which file systems you can choose from now?

What other functions were that?

This is normal, and those are hooks to the formatting utilities installed by NTFS-3G and Tuxera NTFS for Mac, respectively.

As soon as NTFS-3G and Tuxera NTFS for Mac is gone from the system, Disk Utility should be back to normal.
Normally, you uninstall the software from the respective preference panes (System preferences->NTFS-3G and System Preferences->Tuxera NTFS). There's an uninstall button in those that should work.
If that doesn't work for some reason, you can manually clean out the files installed by NTFS-3G by following the instructions in the following thread: http://tuxera.com/forum/viewtopic.php?f=4&t=1271
For Tuxera NTFS for Mac, open up the Terminal (Applications->Utilities->Terminal) and paste the following command:

I hope this helps.
Sorry for the late reply. We've been busy lately. I hope you understand.
Regards,
- Erik

Joined: Sat Oct 06, 2007 09:24
Posts: 530

Re: Can't mount my NTFS disk.

What was the output of 'chkdsk'? Even if it doesn't allow you to do chkdsk on a volume currently in use, is normally possible to schedule a chkdsk at boot time.
You can get more help on chkdsk from Microsoft's support network, for instance: http://support.microsoft.com/kb/315265
Regards,
- Erik