training speed

[WWWJL]

Hello, has TensorNet accelerated during training? Hope you can share the results of the training speed, thank you

[Bihaqo]

In my experience, tensor net topically don't help with the training speed (unless you're going for truly gigantic layers, which would be impractical without tensornet): the time per iteration is usually almost the same (maybe like 10% faster) and you need to do a bit more iterations than when training without tensornet. That's probably because on training a single matmul is rarely the bottleneck of the entire training process.

…

On Thu, 16 Apr 2020 at 07:16, WWWJL ***@***.***> wrote: Hello, has TensorNet accelerated during training? Hope you can share the results of the training speed, thank you — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#7>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABK6V2OD2MDGUCXQLU6T76DRM2PDTANCNFSM4MJHV4JQ> .

[Bihaqo]

Also note that in the MATLAB implementation I wrote the backward pass manually and it's much slower than the speed you can get with autodiff (e.g. by writing it in tensorflow or pytorch) On Thu, 16 Apr 2020 at 13:52, Alexander Novikov <sasha.v.novikov@gmail.com> wrote:

…

In my experience, tensor net topically don't help with the training speed (unless you're going for truly gigantic layers, which would be impractical without tensornet): the time per iteration is usually almost the same (maybe like 10% faster) and you need to do a bit more iterations than when training without tensornet. That's probably because on training a single matmul is rarely the bottleneck of the entire training process. On Thu, 16 Apr 2020 at 07:16, WWWJL ***@***.***> wrote: > Hello, has TensorNet accelerated during training? Hope you can share the > results of the training speed, thank you > > — > You are receiving this because you are subscribed to this thread. > Reply to this email directly, view it on GitHub > <#7>, or unsubscribe > <https://github.com/notifications/unsubscribe-auth/ABK6V2OD2MDGUCXQLU6T76DRM2PDTANCNFSM4MJHV4JQ> > . >