Quantcast

pthreads on 64bit Windows

classic Classic list List threaded Threaded
49 messages Options
123
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: pthreads on 64bit Windows

Orin Eman
On Tue, Jan 26, 2010 at 9:18 PM, Michael Plante <[hidden email]> wrote:

>> >> Even if you add the
>> >> process ID, adding naming pretty much requires you delve into security
>> >> descriptors to avoid people tampering (I think?).
>>
>> I wouldn't worry about that... denial of service would be so easy by just
talking
>> directly to WinUSB or whatever other driver we were using that tinkering
with our
>> semaphore wouldn't be worthwhile!

Are you allowed to talk to WinUSB if you're not an administrator?  (I don't
know; I'm asking.)
 
I'd be in big trouble at work if you can't!
 
>>
>> If you never free the global mutex, I'd be tempted to bypass the
compare/exchange if it isn't null:
>>
>> if ( global_mutex ) {
>>     WaitForSingleObject(global_mutex);
>> } else {
>>     // Create mutex, do the exchange etc.
>> }

I think that's just an optimization.  You still have a race where the mutex
can be created during our "else" by their "else", so you still need the
compare/exchange if it's NULL.  This optimization only adds more things for
us to check for correctness, for minimal gain.  "temp_mut" is not leaked
unless the compare/exchange succeeds, and it only succeds once per process
execution.
 
Right, it's an optimization.
 
 

Ultimately, this all boils down to 2 things:  1) can we tolerate a named
object   2) do others (Nathan, particularly) want the private data.  Both
solutions are simpler than the one I posted, but each has advantages.  Only
if the answer to both of the above is "no" should this method be
implemented.  My answers are "no" and "yes" respectively, though I may yet
be convinced on point #1...
 
My vote would be for private data first.  If not, the named object makes things a little simpler at init time, a little less at exit.  Both ways work and I don't have a real preference.
 
Orin.
 

------------------------------------------------------------------------------
The Planet: dedicated and managed hosting, cloud storage, colocation
Stay online with enterprise data centers and the best network in the business
Choose flexible plans and management services without long-term contracts
Personal 24x7 support from experience hosting pros just a phone call away.
http://p.sf.net/sfu/theplanet-com
_______________________________________________
Libusb-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/libusb-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: pthreads on 64bit Windows

Michael Plante
Orin Eman wrote:
>> Michael Plante wrote:
>> >> Orin Eman wrote:
>> >> >> I wouldn't worry about that... denial of service would be so easy
by just
>> >> >> talking directly to WinUSB or whatever other driver we were using
that tinkering
>> >> >> with our semaphore wouldn't be worthwhile!
>> >>
>> >> Are you allowed to talk to WinUSB if you're not an administrator?  (I
don't
>> >> know; I'm asking.)
>>
>> I'd be in big trouble at work if you can't!

Fair enough.  But, to scale back my argument a bit, there may be different
policies for terminal services (== remote desktop?) users than local users.
Physical access is something to consider:  just unplug the device!
Regardless:


1) I still am not comfortable with introducing new security holes, even if
there are more significant ones that Microsoft caused.  Who knows?  Maybe
they'll clean this one up?  Maybe they can't?  Maybe, in the future,
security descriptors will be attached to devices via udev-like rules (buried
in the bowels of the registry, of course)?

2) During the time your interface is claimed (or whatever the action is that
gives exclusive access), how much is the risk mitigated?  I'd suggest that
the semaphore is vulnerable during, before, and after libusb is used, so
that's a broader window for attacks.

3) Someone can modify libusb (it's LGPL, after all), forget to change the
semaphore name, and create a poorly-behaved program that interferes with a
well-behaved program, creating seemingly-legit support requests for both.

Michael


------------------------------------------------------------------------------
The Planet: dedicated and managed hosting, cloud storage, colocation
Stay online with enterprise data centers and the best network in the business
Choose flexible plans and management services without long-term contracts
Personal 24x7 support from experience hosting pros just a phone call away.
http://p.sf.net/sfu/theplanet-com
_______________________________________________
Libusb-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/libusb-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: pthreads on 64bit Windows

Orin Eman
On Tue, Jan 26, 2010 at 9:56 PM, Michael Plante <[hidden email]> wrote:
Orin Eman wrote:
>> Michael Plante wrote:
>> >> Orin Eman wrote:
>> >> >> I wouldn't worry about that... denial of service would be so easy
by just
>> >> >> talking directly to WinUSB or whatever other driver we were using
that tinkering
>> >> >> with our semaphore wouldn't be worthwhile!
>> >>
>> >> Are you allowed to talk to WinUSB if you're not an administrator?  (I
don't
>> >> know; I'm asking.)
>>
>> I'd be in big trouble at work if you can't!

Fair enough.  But, to scale back my argument a bit, there may be different
policies for terminal services (== remote desktop?) users than local users.
 
 
No difference for terminal services users with default WinUSB settings that I know of.  The only problem I've had is having the device sulk and needing to physically unplug it...
 
 



1) I still am not comfortable with introducing new security holes, even if
there are more significant ones that Microsoft caused.  Who knows?  Maybe
they'll clean this one up?  Maybe they can't?  Maybe, in the future,
security descriptors will be attached to devices via udev-like rules (buried
in the bowels of the registry, of course)?
 
 
 
See http://msdn.microsoft.com/en-us/library/ms794732.aspx for device security descriptors.  You can even specify a security descriptor in the inf file.  Have fun understanding SDDL!
 
 
Orin.
 

------------------------------------------------------------------------------
The Planet: dedicated and managed hosting, cloud storage, colocation
Stay online with enterprise data centers and the best network in the business
Choose flexible plans and management services without long-term contracts
Personal 24x7 support from experience hosting pros just a phone call away.
http://p.sf.net/sfu/theplanet-com
_______________________________________________
Libusb-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/libusb-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: pthreads on 64bit Windows

Michael Plante
Orin Eman wrote:
>> See http://msdn.microsoft.com/en-us/library/ms794732.aspx for device
security
>> descriptors.  You can even specify a security descriptor in the inf file.
Have
>> fun understanding SDDL!

Indeed.  A more complicated security mechanism requires more complicated
configuration.  But big companies will have the resources to either decipher
those in-house or pay for a support contract, and there are probably enough
examples online.  This kinda makes my point, though I was never really
concerned about 1&2 as much as 3, as I run as administrator anyway, and
there's no remote access to my Windows box, other than a few trojans I
installed to keep my life interesting.

If we do use this solution, let's at least append the process ID and make
sure the semaphore or mutex gets destroyed in windows_exit(); that solves
#3, I think.  Ultimately, the first 2 are just concerns other people might
have.  I debug/develop on Windows, but the production target is an embedded
Linux board w/o Internet access and usually surrounded by water on all
sides, so I'm not as concerned about security as this email chain might
imply.

Michael


------------------------------------------------------------------------------
The Planet: dedicated and managed hosting, cloud storage, colocation
Stay online with enterprise data centers and the best network in the business
Choose flexible plans and management services without long-term contracts
Personal 24x7 support from experience hosting pros just a phone call away.
http://p.sf.net/sfu/theplanet-com
_______________________________________________
Libusb-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/libusb-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: pthreads on 64bit Windows

Michael Plante
In reply to this post by Michael Plante
Michael Plante wrote:
>> Yes, it presently uses a mutex.  Like I said, that is not a difficult
problem.  The only
>> issue with the default context is making sure that mutex only gets
overwritten once, and
>> this can be done with an InterlockedCompareExchange()  (are HANDLE's 32
or 64-bits in
>> 64-bit Windows?  If 64, it might need to be
InterlockedCompareExchangePointer().).  Or,
>> if it's done via static initialization, that it get torn down, and that
we check for
>> errors during init.
>>
>> The condition variable is difficult.  I was merely trying to be complete
by listing the others.


Since none of the synchronization objects are exposed in libusb.h, would it
make sense to define some inline functions in libusbi.h that call out to
either Win32 or pthreads, and then create two typedefs:  1 for standalone
mutexes, and 1 for a struct containing a mutex and a condition variable?
(The reason for the struct would be because it may be something completely
different inside the struct in Win32.)  For the mutexes (even the static
one), the preprocessor output should be identical to the old code, and for
the condition variable, it'll only be slightly different.

The primitives needing to be replaced are (assuming we ignore
dpfp_threaded.c) :

pthread_mutex_init/pthread_mutex_destroy
pthread_mutex_lock/pthread_mutex_unlock
pthread_mutex_trylock
pthread_cond_init /pthread_cond_destroy
pthread_cond_broadcast
pthread_cond_wait
pthread_cond_timedwait

PTHREAD_MUTEX_INITIALIZER (used only for the default context lock)

pthread_mutex_t
pthread_cond_t


What I would suggest for standalone mutexes is along the lines of:

#ifdef OS_WINDOWS
#define libusb_mutex HANDLE
#else
#define libusb_mutex pthread_mutex_t
#endif

#ifdef OS_WINDOWS
static int _inline libusb_mutex_init(libusb_mutex *mutex, void const *attr)
{
        if(! mutex) return ((errno=EINVAL));
        *mutex = CreateMutex(NULL, FALSE, NULL);
        if(!*mutex) return ((errno=ENOMEM));
        return 0;
}
#else
static int  inline libusb_mutex_init(libusb_mutex_t *restrict mutex,
                              const pthread_mutexattr_t *restrict attr) {
        return pthread_mutex_init(mutex, attr);
}
#endif


#ifdef OS_WINDOWS
#define PTHREAD_MUTEX_INITIALIZER NULL
static int _inline libusb_mutex_static(libusb_mutex *mutex) {
        libusb_mutex temp_mutex; // N.B.: (opaque) pointer; stack is ok
        if (libusb_mutex_init(&temp_mutex, NULL))
                return 0; // fail
        if (InterlockedCompareExchangePointer(
                        mutex, temp_mutex, NULL)) {
                libusb_mutex_destroy(&temp_mutex);
        }
        return 1; // succeed
}
#else
static int  inline libusb_mutex_static(libusb_mutex *mutex) {
        return 1; // don't have to fake it.
}
#endif


#ifdef OS_WINDOWS
static int _inline libusb_mutex_lock(libusb_mutex_t *mutex) {
      DWORD result;
        if(!mutex) return ((errno=EINVAL));
        result = WaitForSingleObject(*mutex, INFINITE);
        if(result == WAIT_OBJECT_0 || result == WAIT_ABANDONED)
                return 0; // acquired (abandoned is bad, but I don't know mapping)
        return ((errno=EINVAL)); // don't know how this would happen, so don't know
proper errno
}
#else
static int  inline libusb_mutex_lock(libusb_mutex_t *mutex) {
        return pthread_mutex_lock(mutex);
}
#endif


In libusb_init():
+ if(!libusb_mutex_static(&default_context_lock))
+ goto err;
        pthread_mutex_lock(&default_context_lock);



...and so on.  No change is needed to the static definition.  Again, this
leaks one mutex per process execution, but for many programs, that's a
difference of milliseconds between when it's freed by the OS and when it
could be freed by libusb_exit(), and a small known-constant # is acceptable,
IMHO.  I didn't try compiling these, so there may be problems, but I think
it gets the idea across.

I want to be careful about pthread_* name collisions, since when I use
libusb on Win32, I already have a pthread dependency for unrelated reasons.
Maybe it's not an issue...

Thoughts?

Thanks,
Michael




------------------------------------------------------------------------------
The Planet: dedicated and managed hosting, cloud storage, colocation
Stay online with enterprise data centers and the best network in the business
Choose flexible plans and management services without long-term contracts
Personal 24x7 support from experience hosting pros just a phone call away.
http://p.sf.net/sfu/theplanet-com
_______________________________________________
Libusb-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/libusb-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: pthreads on 64bit Windows

francescomontorsi
In reply to this post by Pete Batard
Hi,

2010/1/27 Pete Batard <[hidden email]>:
> Hi Francesco,
>
>> could anyone who has the 64bit build of pthread-win32 upload it
>> somewhere or send it to me :) ? Thanks in advance!!
>
> http://libusb-winusb-wip.googlecode.com/files/pthread-win32_x64.zip
> Sorry it took so long to upload it.
Thank you very much!

In the past days I've successfully created the WinUSB-based driver for
a project I've ported to libusb-1.0.
Now I managed to find a MSVC++ 2008 Professional edition and used it
to compile in 64bit-mode all dependencies of that project (namely
wxWidgets, libusb-1.0).
Unfortunately I'm having an hard time trying to debug a heap
corruption assert which is popped-up semi-randomly by the final
application.

I think one possible cause of the problem could be that I'm linking in
the final EXE libraries compiled with different /MDd or /MTd
settings... so here's my new question: which setting (/MDd or /MTd or
something else?) was used to compile the pthreadVC2_x64.dll ? ;)

I have noticed that the msvc2008 project of libusb-1.0 uses /MTd and
I've changed it to /MDd (which btw is the default used by wxWidgets)
since I think it's better to use the DLL version of the CRT (see
http://msdn.microsoft.com/en-us/library/abx4dbyh%28VS.71%29.aspx)...
was /MTd chosen for some particular reason?

Thanks!!
Francesco

------------------------------------------------------------------------------
The Planet: dedicated and managed hosting, cloud storage, colocation
Stay online with enterprise data centers and the best network in the business
Choose flexible plans and management services without long-term contracts
Personal 24x7 support from experience hosting pros just a phone call away.
http://p.sf.net/sfu/theplanet-com
_______________________________________________
Libusb-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/libusb-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: pthreads on 64bit Windows

Pete Batard
On 2010.01.30 12:23, Francesco wrote:
> In the past days I've successfully created the WinUSB-based driver for
> a project I've ported to libusb-1.0.

Great.

> I think one possible cause of the problem could be that I'm linking in
> the final EXE libraries compiled with different /MDd or /MTd
> settings... so here's my new question: which setting (/MDd or /MTd or
> something else?) was used to compile the pthreadVC2_x64.dll ? ;)

The pthread-win32 I compiled are using /MD. I can produce /MT ones if
required.

> I have noticed that the msvc2008 project of libusb-1.0 uses /MTd and
> I've changed it to /MDd (which btw is the default used by wxWidgets)
> since I think it's better to use the DLL version of the CRT (see
> http://msdn.microsoft.com/en-us/library/abx4dbyh%28VS.71%29.aspx)...
> was /MTd chosen for some particular reason?

On 2010.01.20 19:24, Orin Eman wrote:
 > Last I saw, the DLL was using the DLL C runtime.  I'd strongly recommend
 > using the (multi-threaded) static C runtime, otherwise, binary
 > distributions would require the VC redistributables (and we start
 > worrying about side by side configurations and so on).

I don't really mind either way, but if this is going to be an issue with
pthread-win32, I'd rather pick the one that produces the less headache.

Of course, we will remove pthread-win32 use after the first version is
out, so that problem should soon be moot.

Oh, and if you're having issues with threading, you might want to test
with the "concurency" branch instead, which is available at:
http://git.libusb.org/?p=libusb-pbatard.git;a=shortlog;h=refs/heads/concurrency
This branch removes the need for pthread in the Windows specific source
(but not on core), so it might help.

Regards,

/Pete

------------------------------------------------------------------------------
The Planet: dedicated and managed hosting, cloud storage, colocation
Stay online with enterprise data centers and the best network in the business
Choose flexible plans and management services without long-term contracts
Personal 24x7 support from experience hosting pros just a phone call away.
http://p.sf.net/sfu/theplanet-com
_______________________________________________
Libusb-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/libusb-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: pthreads on 64bit Windows

Michael Plante
Pete Batard wrote:
>> The pthread-win32 I compiled are using /MD. I can produce /MT ones if
>> required.

For prebuilt binaries, that's probably a good idea.  We are providing a
"sources" file for libusb, which allows Francesco to change to /MD* there,
but not in pthread.


>> Francesco wrote:
>> > I have noticed that the msvc2008 project of libusb-1.0 uses /MTd and
>> > I've changed it to /MDd (which btw is the default used by wxWidgets)
>> > since I think it's better to use the DLL version of the CRT (see
>> > http://msdn.microsoft.com/en-us/library/abx4dbyh%28VS.71%29.aspx)...
>> > was /MTd chosen for some particular reason?

Did doing that to pthreads stop the corruption errors, since you now have
access to a system that should build it?


>> On 2010.01.20 19:24, Orin Eman wrote:
>> > Last I saw, the DLL was using the DLL C runtime.  I'd strongly
recommend
>> > using the (multi-threaded) static C runtime, otherwise, binary
>> > distributions would require the VC redistributables (and we start
>> > worrying about side by side configurations and so on).
>>
>> I don't really mind either way, but if this is going to be an issue with
>> pthread-win32, I'd rather pick the one that produces the less headache.
>>
>> Of course, we will remove pthread-win32 use after the first version is
>> out, so that problem should soon be moot.

So I take it the code I posted to the list for this is not a direction we
want to follow anytime soon?


>> Oh, and if you're having issues with threading, you might want to test
>> with the "concurency" branch instead, which is available at:

It might be threading, but Francesco did not mention threads.  It might also
just be a matter of, as we discussed, having multiple copies of the CRT
floating around.


>>
http://git.libusb.org/?p=libusb-pbatard.git;a=shortlog;h=refs/heads/concurre
ncy
>> This branch removes the need for pthread in the Windows specific source
>> (but not on core), so it might help.

Were you considering merging this into master?

Thanks,
Michael


------------------------------------------------------------------------------
The Planet: dedicated and managed hosting, cloud storage, colocation
Stay online with enterprise data centers and the best network in the business
Choose flexible plans and management services without long-term contracts
Personal 24x7 support from experience hosting pros just a phone call away.
http://p.sf.net/sfu/theplanet-com
_______________________________________________
Libusb-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/libusb-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: pthreads on 64bit Windows

Pete Batard
On 2010.01.30 17:58, Michael Plante wrote:
>>> The pthread-win32 I compiled are using /MD. I can produce /MT ones if
>>> required.
>
> For prebuilt binaries, that's probably a good idea.  We are providing a
> "sources" file for libusb, which allows Francesco to change to /MD* there,
> but not in pthread.

Yeah. I'll produce /MT ones and upload them when I get a chance.

>>> Of course, we will remove pthread-win32 use after the first version is
>>> out, so that problem should soon be moot.
>
> So I take it the code I posted to the list for this is not a direction we
> want to follow anytime soon?

I'd say "anytime soon" depends on when we have a first official release.
As I mentioned privately, I'm reluctant to add threading changes to core
with the first release, even if they are trivial, because we are
introducing enough modifications as it is (I'll be sending the core
patches soon).

> Were you considering merging (the concurrency branch) into master?

It *will* be merged into master before the first release. In fact, I've
been integrating changes from master into concurrency so that it can be
merged back with master. I was just waiting to see if there was anything
more that needed to be done in concurrency, but I'd say it's probably
stable enough now.

Regards,

/Pete

------------------------------------------------------------------------------
The Planet: dedicated and managed hosting, cloud storage, colocation
Stay online with enterprise data centers and the best network in the business
Choose flexible plans and management services without long-term contracts
Personal 24x7 support from experience hosting pros just a phone call away.
http://p.sf.net/sfu/theplanet-com
_______________________________________________
Libusb-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/libusb-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: pthreads on 64bit Windows

francescomontorsi
In reply to this post by Michael Plante
Hi all,
   thanks for your info.

I've debugged the problem and actually there were 2 things to consider:

1) until I didn't switch from /MTd to /MDd in the MSVC project that
I'm now using to build libusb-1.0, the heap problems were very
probably caused by the conflicting CRT settings (as I said wxWidgets
uses /MDd).
Before realizing this, to help me in the process of debugging the heap
corruption I started using the "AppVerifier" tool from MS (see
http://www.microsoft.com/downloads/details.aspx?displaylang=en&FamilyID=c4a25ab9-649d-4a1b-b4a7-c9d8b095df18).
AppVerifier helps debugging heap problems (and many other types) by
injecting some code in the application which e.g. checks the
heap/stack after each malloc/free call.
However it turned out to be misleading:

2) after fixing the /MTd => /MDd setting, I found that the application
was stopping in the CancelIo calls done by _free_index() (which was
triggered by handle_timeout() calls) with a message like:
     First-chance exception at 0x7794fec7 in usbpicprog.exe:
0xC0000008: An invalid handle was specified.
It took me a bit of time to find out that CancelIo() is called on
possibly-invalid handles by purpose and that the exception above was
generated by AppVerifier.

Now, after disabling AppVerifier and with all libs using /MDd (except
for pthread-win32 with /MD) I don't have any heap-corruption problem.
The application works but unfortunately I get rather often "time out"
errors when calling libusb_bulk_transfer() for read or write.
I'm now investigating these errors (they didn't show up with libusb-0.1)...


2010/1/30 Michael Plante <[hidden email]>:
> Pete Batard wrote:
>>> The pthread-win32 I compiled are using /MD. I can produce /MT ones if
>>> required.
ok, thanks.

>>> Francesco wrote:
>>> > I have noticed that the msvc2008 project of libusb-1.0 uses /MTd and
>>> > I've changed it to /MDd (which btw is the default used by wxWidgets)
>>> > since I think it's better to use the DLL version of the CRT (see
>>> > http://msdn.microsoft.com/en-us/library/abx4dbyh%28VS.71%29.aspx)...
>>> > was /MTd chosen for some particular reason?
>
> Did doing that to pthreads stop the corruption errors, since you now have
> access to a system that should build it?
I didn't try to recompile pthreads myself: I think mixing /MD and /MDd is fine.
The problems probably arise when mixing /MT and /MD switches...


>>> On 2010.01.20 19:24, Orin Eman wrote:
>>> > Last I saw, the DLL was using the DLL C runtime.  I'd strongly
> recommend
>>> > using the (multi-threaded) static C runtime, otherwise, binary
>>> > distributions would require the VC redistributables (and we start
>>> > worrying about side by side configurations and so on).
>>>
>>> I don't really mind either way, but if this is going to be an issue with
>>> pthread-win32, I'd rather pick the one that produces the less headache.
>>>
>>> Of course, we will remove pthread-win32 use after the first version is
>>> out, so that problem should soon be moot.
great! looking forward for it ;)


>>> Oh, and if you're having issues with threading, you might want to test
>>> with the "concurency" branch instead, which is available at:
>
> It might be threading, but Francesco did not mention threads.
actually the program I'm developing initializes the USB device from
its primary thread but then makes long read/write operations from a
secondary thread, so indeeed multithreading could be a source of
error.
I'll eventually test the concurrency branch...

Thanks,
Francesco

------------------------------------------------------------------------------
The Planet: dedicated and managed hosting, cloud storage, colocation
Stay online with enterprise data centers and the best network in the business
Choose flexible plans and management services without long-term contracts
Personal 24x7 support from experience hosting pros just a phone call away.
http://p.sf.net/sfu/theplanet-com
_______________________________________________
Libusb-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/libusb-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: pthreads on 64bit Windows

Michael Plante
Francesco wrote:
>> Before realizing this, to help me in the process of debugging the heap
>> corruption I started using the "AppVerifier" tool from MS (see
>>
http://www.microsoft.com/downloads/details.aspx?displaylang=en&FamilyID=c4a2
5ab9-649d-4a1b-b4a7-c9d8b095df18).
>> AppVerifier helps debugging heap problems (and many other types) by
>> injecting some code in the application which e.g. checks the
>> heap/stack after each malloc/free call.
>> However it turned out to be misleading:


One other option you can try, if you're interested in checking the heap, is
sticking this at the start of main() (or before you do any allocations):

_CrtSetDbgFlag(_CRTDBG_ALLOC_MEM_DF|/*_CRTDBG_CHECK_ALWAYS_DF|_CRTDBG_DELAY_
FREE_MEM_DF|*/_CRTDBG_LEAK_CHECK_DF);

and remember to include <crtdbg.h> .  You can read the docs on what the
commented flags do, but the ones I left in don't seem to have much
performance impact.  Then, when you run in the debugger, it'll dump leaks
when the program exits.  Some of those flags also check before/after
malloc/free, but I don't remember which ones.  Some of the builtin CRT code
exempts its own allocations from this checking, which is why you can get
away with putting it in main(), rather than digging into the CRT code that
calls main().


HTH,
Michael


------------------------------------------------------------------------------
The Planet: dedicated and managed hosting, cloud storage, colocation
Stay online with enterprise data centers and the best network in the business
Choose flexible plans and management services without long-term contracts
Personal 24x7 support from experience hosting pros just a phone call away.
http://p.sf.net/sfu/theplanet-com
_______________________________________________
Libusb-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/libusb-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: pthreads on 64bit Windows

Pete Batard
In reply to this post by Pete Batard
On 2010.01.30 20:47, Pete Batard wrote:
>> Were you considering merging (the concurrency branch) into master?
>
> It *will* be merged into master before the first release.

Concurrency branch now fully merged into master (r128).

Note that with r127, the timer functions are now handled by a separate
thread (with its affinity set to the first core), so libusb on Windows
is now running at least 2 separate threads.

Regards,

/Pete

------------------------------------------------------------------------------
The Planet: dedicated and managed hosting, cloud storage, colocation
Stay online with enterprise data centers and the best network in the business
Choose flexible plans and management services without long-term contracts
Personal 24x7 support from experience hosting pros just a phone call away.
http://p.sf.net/sfu/theplanet-com
_______________________________________________
Libusb-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/libusb-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: pthreads on 64bit Windows

Michael Plante
In reply to this post by Pete Batard
Pete Batard wrote:
>> I'd say "anytime soon" depends on when we have a first official release.
>> As I mentioned privately, I'm reluctant to add threading changes to core
>> with the first release, even if they are trivial, because we are
>> introducing enough modifications as it is (I'll be sending the core
>> patches soon).

If you want, I can put up a new branch and start making those changes there.
I was hoping to hear from Peter or Daniel whether they're clean enough.  In
any case, I can't do it until Monday night (UTC-6 here).

Michael


------------------------------------------------------------------------------
The Planet: dedicated and managed hosting, cloud storage, colocation
Stay online with enterprise data centers and the best network in the business
Choose flexible plans and management services without long-term contracts
Personal 24x7 support from experience hosting pros just a phone call away.
http://p.sf.net/sfu/theplanet-com
_______________________________________________
Libusb-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/libusb-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: pthreads on 64bit Windows

Pete Batard
On 2010.01.31 00:19, Michael Plante wrote:
> If you want, I can put up a new branch and start making those changes there.

I'm fine with that. In fact, I'll probably mirror your branch as soon as
you create it, as these changes are definitely going to be part of the
2nd official release.

Regards,

/Pete

------------------------------------------------------------------------------
The Planet: dedicated and managed hosting, cloud storage, colocation
Stay online with enterprise data centers and the best network in the business
Choose flexible plans and management services without long-term contracts
Personal 24x7 support from experience hosting pros just a phone call away.
http://p.sf.net/sfu/theplanet-com
_______________________________________________
Libusb-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/libusb-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: pthreads on 64bit Windows

francescomontorsi
In reply to this post by Michael Plante
Hi Michael!

2010/1/30 Michael Plante <[hidden email]>:
> One other option you can try, if you're interested in checking the heap, is
> sticking this at the start of main() (or before you do any allocations):
>
> _CrtSetDbgFlag(_CRTDBG_ALLOC_MEM_DF|/*_CRTDBG_CHECK_ALWAYS_DF|_CRTDBG_DELAY_
> FREE_MEM_DF|*/_CRTDBG_LEAK_CHECK_DF);
>
> and remember to include <crtdbg.h> .
thanks for the tip!
I'm going to add this call to my main() asap!

Thanks,
Francesco

------------------------------------------------------------------------------
The Planet: dedicated and managed hosting, cloud storage, colocation
Stay online with enterprise data centers and the best network in the business
Choose flexible plans and management services without long-term contracts
Personal 24x7 support from experience hosting pros just a phone call away.
http://p.sf.net/sfu/theplanet-com
_______________________________________________
Libusb-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/libusb-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: pthreads on 64bit Windows

Graeme Gill
In reply to this post by Pete Batard
Pete Batard wrote:
> Note that with r127, the timer functions are now handled by a separate
> thread (with its affinity set to the first core), so libusb on Windows
> is now running at least 2 separate threads.

Sorry if this discussion has been had before, but can I
ask a dumb question: Why is a timer needed at all, and
why does it need to be high precision ?

libusb 0.1 doesn't have OS specific timing structures
like timespec visible in the API and doesn't deal with
absolute time anyway, and the OS backends deal with the
system specific issues. Why isn't libusb 1.0 taking the same approach ?

Do you really need better than 1msec precision on this anyway ?
(ie. why aren't the win32 primitives that take timeouts in msec
+ timeGetTime() sufficient for MSWindows back end ?)

Graeme Gill.

------------------------------------------------------------------------------
The Planet: dedicated and managed hosting, cloud storage, colocation
Stay online with enterprise data centers and the best network in the business
Choose flexible plans and management services without long-term contracts
Personal 24x7 support from experience hosting pros just a phone call away.
http://p.sf.net/sfu/theplanet-com
_______________________________________________
Libusb-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/libusb-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: pthreads on 64bit Windows

Michael Plante
Graeme Gill wrote:

>> Pete Batard wrote:
>> > Note that with r127, the timer functions are now handled by a separate
>> > thread (with its affinity set to the first core), so libusb on Windows
>> > is now running at least 2 separate threads.
>>
>> Sorry if this discussion has been had before, but can I
>> ask a dumb question: Why is a timer needed at all, and
>> why does it need to be high precision ?
>>
>> libusb 0.1 doesn't have OS specific timing structures
>> like timespec visible in the API and doesn't deal with
>> absolute time anyway, and the OS backends deal with the
>> system specific issues. Why isn't libusb 1.0 taking the same approach ?

Well, looking at libusb-win32 0.1, anyway, the timeouts are relative, but
they're relative to when you try to reap in the async api
(usb_reap_async()).  In the sync api, you don't have more than one going on
from a given thread, so relative is fine.  So I believe that the reason for
absolute times is to have multiple operations going.

As to *accuracy* (forget precision for a moment), yes, we did have this
discussion, and Tim has seen up to 0.5 seconds of skew between processors on
the HF performance counter:

http://old.nabble.com/Re%3A--PATCH%3A-winusb--hires-timer-fix-p26749357.html

The reason for the thread is so we can lock it to one processor without
screwing up all the threads in an app.  I expect some delay to be introduced
in the context switches to get the time, but I think it's better than what
we had.  AFAIK, we really only need the thread for the monotonic clock, not
the realtime clock, right?  Maybe we can simplify/optimize things there.  Is
there any skew in GetSystemTimeAsFileTime?  However, most of the time the
monotonic clock IS what's used, except libusb_wait_for_event(), which is for
pthread_cond_timedwait (of course, depending on how CV's are emulated, we
may get away with a relative timeout there, but that obviously doesn't solve
the timer thread problem).


>> Do you really need better than 1msec precision on this anyway ?
>> (ie. why aren't the win32 primitives that take timeouts in msec
>> + timeGetTime() sufficient for MSWindows back end ?)

Hrm.  I don't recall if we've discussed that option.  timeGetTime() sounds
monotonic if your system can't stay up 50 days, which is a good bet on
Windows... :)  And we could mess around with it so the only requirement is
that you have to call into libusb at least every 50 days (update a
high-order DWORD counter).  The docs don't talk about XP/later, but they say
on NT/2k, the accuracy can be worse than 5ms in some cases, and that there
are functions that *might* work which will lower that granularity.  I don't
know if any of this is a good idea.

Michael


------------------------------------------------------------------------------
The Planet: dedicated and managed hosting, cloud storage, colocation
Stay online with enterprise data centers and the best network in the business
Choose flexible plans and management services without long-term contracts
Personal 24x7 support from experience hosting pros just a phone call away.
http://p.sf.net/sfu/theplanet-com
_______________________________________________
Libusb-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/libusb-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: pthreads on 64bit Windows

Graeme Gill
Michael Plante wrote:
> Well, looking at libusb-win32 0.1, anyway, the timeouts are relative, but
> they're relative to when you try to reap in the async api
> (usb_reap_async()).  In the sync api, you don't have more than one going on
> from a given thread, so relative is fine.  So I believe that the reason for
> absolute times is to have multiple operations going.

I've had a look at some of the doco but I can't say I'm understanding
it. There is the statement:

"* - libusb also needs to be called into at certain fixed points in time in
  * order to accurately handle transfer timeouts."

but this seems to me to be complicating the API unnecessarily to cope
with certain platforms. libusb 0.1 didn't have these complications,
hiding platform dependencies in the implementation. The libusb 1.0
async scheme seems similar to MSWindows OVERLAPPED type approach,
but it doesn't need absolute time, so I'm struggling to understand
why libusb 1.0 does. Don't you kick off an operation with a given
timeout, and then get called back when it completes or times out ?
Why do you have to poll it ? Why should different operations in the
same thread interfere with each other ?

> As to *accuracy* (forget precision for a moment), yes, we did have this
> discussion, and Tim has seen up to 0.5 seconds of skew between processors on
> the HF performance counter:

Right, but why use them if you don't need too given the complication
of needing a thread with affinity ?

> Hrm.  I don't recall if we've discussed that option.  timeGetTime() sounds
> monotonic if your system can't stay up 50 days, which is a good bet on
> Windows... :)

It's supposed to be monotonic if you use the recommended 32 bit
modulo arithmetic, even over and after 50 days. Since MSWindows
timeout API's all take values in msec, and given the multi-core
and speed throttling issues with the HP timers, I would guess this
is what they are using internally

Graeme Gill.

------------------------------------------------------------------------------
The Planet: dedicated and managed hosting, cloud storage, colocation
Stay online with enterprise data centers and the best network in the business
Choose flexible plans and management services without long-term contracts
Personal 24x7 support from experience hosting pros just a phone call away.
http://p.sf.net/sfu/theplanet-com
_______________________________________________
Libusb-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/libusb-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: pthreads on 64bit Windows

Michael Plante
Graeme Gill wrote:
>> Michael Plante wrote:
>> > Well, looking at libusb-win32 0.1, anyway, the timeouts are relative,
but
>> > they're relative to when you try to reap in the async api
>> > (usb_reap_async()).  In the sync api, you don't have more than one
going on
>> > from a given thread, so relative is fine.  So I believe that the reason
for
>> > absolute times is to have multiple operations going.
>>
>> I've had a look at some of the doco but I can't say I'm understanding
>> it. There is the statement:
>>
>> "* - libusb also needs to be called into at certain fixed points in time
in
>>   * order to accurately handle transfer timeouts."
>>
>> but this seems to me to be complicating the API unnecessarily to cope
>> with certain platforms. libusb 0.1 didn't have these complications,

Because the timeouts were referenced to when the reap was attempted, not
when you submitted the transfer, which can be at very different times (the
former later).  The latter is the logical place to reference the timeout to,
but libusb only did this with the sync interface.  I don't see much
platform-dependent about it.  Would you do it differently?  Certainly it
could be simpler if we only supported sync.


>> hiding platform dependencies in the implementation. The libusb 1.0
>> async scheme seems similar to MSWindows OVERLAPPED type approach,

I believe overlapped relies on something in the kernel to set an event at
the appropriate time.  Ultimately, poll does use a relative time, but it has
the old relative time, minus whatever time has elapsed since you first set
the timeout.  This last part is, I believe, key.


>> but it doesn't need absolute time, so I'm struggling to understand
>> why libusb 1.0 does. Don't you kick off an operation with a given
>> timeout, and then get called back when it completes or times out ?

Not sure.  I would assume that, in order to get called back, something in
libusb has to be running first, right?


>> Why do you have to poll it ?

Well, it's not busy-waiting; your thread is suspended.  Assuming you knew
that, see the previous paragraph.


>> Why should different operations in the
>> same thread interfere with each other ?

I don't follow.  Could you clarify?


>> Right, but why use them if you don't need too given the complication
>> of needing a thread with affinity ?

Agreed, *if* you don't need to.  I'm not sure.


>> It's supposed to be monotonic if you use the recommended 32 bit
>> modulo arithmetic, even over and after 50 days.

I don't know that the libusb core does use modulo arithmetic, though it
probably could.  That was what I meant when I said "mess with".  That still
requires you call into libusb often enough to increment an overflow counter,
or call in often enough that it doesn't matter.  Point being, if the
difference exceeds 50 days, you have a problem.


>> Since MSWindows
>> timeout API's all take values in msec, and given the multi-core
>> and speed throttling issues with the HP timers, I would guess this
>> is what they are using internally

They may boil down to the same system call (don't know), but the scheduler
probably doesn't use mm* functions.

Michael


------------------------------------------------------------------------------
The Planet: dedicated and managed hosting, cloud storage, colocation
Stay online with enterprise data centers and the best network in the business
Choose flexible plans and management services without long-term contracts
Personal 24x7 support from experience hosting pros just a phone call away.
http://p.sf.net/sfu/theplanet-com
_______________________________________________
Libusb-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/libusb-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: pthreads on 64bit Windows

Michael Plante
In reply to this post by Pete Batard
Pete Batard wrote:
>> On 2010.01.31 00:19, Michael Plante wrote:
>> > If you want, I can put up a new branch and start making those changes
there.
>>
>> I'm fine with that. In fact, I'll probably mirror your branch as soon as
>> you create it, as these changes are definitely going to be part of the
>> 2nd official release.

Ok, the branch is up.  Please help me test it.  I have not yet made CV
changes, and I have left the mutex associated with the CV using a
pthread_mutex_t.  There is a #define USE_PTHREAD in libusbi.h and
windows_compat.h .  The comments should explain it.  1&2 work.  Once 0
works, 1 will be deprecated.  2 should behave identically to before, while 1
is the "new stuff".

http://git.libusb.org/?p=libusb-mplante.git;a=shortlog;h=refs/heads/lesspthr
ead

Michael


------------------------------------------------------------------------------
The Planet: dedicated and managed hosting, cloud storage, colocation
Stay online with enterprise data centers and the best network in the business
Choose flexible plans and management services without long-term contracts
Personal 24x7 support from experience hosting pros just a phone call away.
http://p.sf.net/sfu/theplanet-com
_______________________________________________
Libusb-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/libusb-devel
123
Loading...