Problem with watchify recipe

[rikukissa]

It seems that if you create a task similar to the "watch" task in watchify recipe gulp.dest is not emitting "end" event in some cases. This prevents you from continuing the pipeline for example with gulp-livereload.

Everything seems to be working fine when you run the task for the first time, or if you save a file that's required by the entry file. When you save the entry file watchify emits the "update" event, the rebundle function is called and bundle file is written, but the pipeline seems to get stuck to gulp.dest and everything after that is ignored.

Also everything seems to be working fine if you have no require calls on your entry file.

I was able to track it down to vinyl-fs library's writeStream method, where for some reason the "finish" listener is not triggered on line 15.

Here's my task

[laurelnaiad]

Interesting... I just came to a point where I'm wondering whether gulp.src is also not emitting end events. Here's what I see.

If I call gulp.src once in a task with a glob that produces no files, then call it again in the same pipeline, the stream that gulp.src is building up with actual files suddenly "goes dark". I've narrowed it down to the following:

https://github.com/wearefractal/vinyl-fs/blob/master/lib/src/index.js#L33
the tail end of the gulp.src module...

  var stream = globStream
    .pipe(formatStream)
    .pipe(getStats(opt));

  if (!opt.read) return stream; // no reading required

/******************************
the stream object is fine -- it has files to pipe.  If I run my own map function here I can log their paths
******************************/

/******************************
but the following just doesn't "do" anything
.... see below for what it does "do"
*****************************/
  return stream.pipe(getContents(opt));
};

https://github.com/wearefractal/vinyl-fs/blob/master/lib/src/getContents.js#L7:

var map = require('map-stream');

var readDir = require('./readDir');
var bufferFile = require('./bufferFile');
var streamFile = require('./streamFile');

module.exports = function(opt) {
/***************************
I can console.log here and I see it
***************************/
  return map(function (file, cb) {
/*************************
THIS map FUNCTION IS NEVER ENTERED
*************************/
    // don't fail to read a directory
    if (file.stat && file.stat.isDirectory()) {
      return readDir(file, cb);
    }

    // read and pass full contents
    if (opt.buffer) {
      return bufferFile(file, cb);
    }

    // dont buffer anything - just pass streams
    return streamFile(file, cb);
  });
};

I've been scratching my head on this for a couple of hours. My latest theory is that an end call somewhere along the line may have been neglected... and so that the second time around something might not be in a fresh state... or maybe an extra end call somewhere.. I'm really puzzled. It's like the stream suddenly goes dark for that module.

In your case, I wonder if the finish method isn't running because it's behind a .once... and has already happened once :) . I've also wondered whether it's an OS timing issue withe file i/o.

This is my failing snippet:

    require('vinyl-fs').src('someNonExistentDir', {cwd: __dirname, read: false})
    .pipe(    require('vinyl-fs').src(['/someGlobWithMatches/**/*'], {cwd: __dirname}))
// there should be files in the stream now but there aren't

So you see why I might suspect a forgotten end call... the theory being that if the first call to src doesn't have anything to stream over, it might "forget" to call end at the end of the non-existent data. Why it would manifest itself in the particular line of code that it does is a mystery to me.

[yocontra]

@stu-salsbury That is definitely a bug with the stream queueing system. It should emit end when no items are found. Can you file this on the vinyl-fs repo?

[laurelnaiad]

I will absolutely if I can prove my theory. So far the theory isn't proving out when I dig into vinyl-fs and glob-stream, so it may be something else. My latest suspect is map-stream, but something tells me that's been used enough so as to be an unlikely location to find an error. So basically, I don't know where the problem is coming from any more. I was actually leaning toward implementing your plans to make .src additive, which I'm pretty sure would make this issue disappear, even if some underlying code still had a bug... I guess one way or the other it isn't gulp code that's responsible here, so I will certainly take the issue down to wherever it is that I find a place to either blame/fix.

[laurelnaiad]

I put some tests in a branch and raised an issue: gulpjs/vinyl-fs#16

[mtscout6]

This conversation alone seems to me like it validates the need for a plugin. I am not a fan of saying "hey copy and paste this code that may change over time into your project". That sounds to me like a need to package up the code, provide sensible extensibility points and update that source as needed.

I also don't feel like the example is good at showing how you would setup your watch task (for dev time only) and your build task. This would easily result in newcomers using different workflows for build vs watch time work. This separation of workflow may result in unexpected differences between dev and production builds.

For this reason I have found that the existing gulp-watchify plugin is moving in the right direction. It still has it's kinks which I'm trying to work out (See: marcello3d/gulp-watchify#4) It ascribes to the DRY model better than a published recipe.

[yocontra]

@mtscout6 I agree with you, I'm not a fan of the current implementation at all. If a good gulp-browserify or gulp-watchify plugin were to exist, I would give it a thumbs up. The current browserify plugin is a big and buggy monster though, which is why it's blacklisted.

[mtscout6]

That's a fair reason to black list it; I'm just worried that the stance I've been seeing from the GulpJs team is that the plugin is not necessary. It's comments like this that concern me:

Watchify doesn't have a gulp plugin, but it doesn't need one either - Gulp recipes

[yocontra]

@mtscout6 Let's focus on browserify first. watchify is another issue since it technically never ends, I'm not sure that makes a good plugin. I understand that browserify is slow as shit without watchify because it doesn't do any caching, but I think we could figure out a way to solve that. watchify is kind of a hack IMO to work around this browserify flaw

[yocontra]

See also https://stackoverflow.com/questions/16275325/node-js-browserify-slow-isnt-there-a-way-to-cache-big-libraries

[mtscout6]

I'm all about getting a better plugin in place. Which means we need to start building that plugin somewhere; instead of prescribing copy paste recipes.

[yocontra]

@mtscout6 You don't need to sell us, we all agree. The recipe is a stopgap so people quit using gulp-browserify. You don't need to keep arguing. If you want to make a better plugin go ahead and I'll tweet it out and link it in all relevant issues. I don't have time to do it, so somebody else needs to.

[yocontra]

Is this still an issue or can I close this? The new watchify and browserify support caching and are much faster BTW

[thealexbaron]

Does this issue seem like it's related to this bug?

http://stackoverflow.com/questions/27260066/gulp-with-watchify-browserify-runs-twice-then-stops-watching

[rs3d]

I've got the same problem with watchify 2.2.1.

To overcome this my gulpfile is like following:

var isWatching = false; // default

var  browserify = require('browserify'),
    watchify = require('watchify'),
    transform = require('vinyl-transform');

gulp.task('watch', ['browser-sync'], function() {
    isWatching = true; // is set when running: gulp watch
    gulp.watch(assetsSrc + 'js/**', ['scripts']);
});

gulp.task('scripts', function(cb) {
    var browserified = transform(function(filename) {
        var b = browserify(filename, {
            debug: true,
            paths: [assetsSrc + 'js', './bower_components', './node_modules']
        });
        return b.bundle();
    });

    var watchified = transform(function(filename) {
        var options = watchify.args;
        options.debug = true;
        options.paths = [assetsSrc + 'js', './bower_components', './node_modules'];
        var w = watchify(browserify(filename, options));
        return w.bundle();
    });

    var buildFn = isWatching ? watchified : browserified;

    return gulp.src([
            assetsSrc + 'js/*.js',
        ])
        .pipe(buildFn)
        .on('error', errorHandler)
        .pipe(gulp.dest(assetsDev + 'js'))
        .pipe(reload({
            stream: true
        }));

});

So my script task decides whether or not to use watchify by the variable isWatching.
if (isWatching === false) the "normal" browserify functions is used.