Skip to content

model放好了 run_model 出现这个错误 把run_mode.pyl里的batch_size值改小 还是报这个错 #8

@lvZic

Description

@lvZic

2018-03-13 15:20:04.122000: I C:\tf_jenkins\workspace\rel-win\M\windows-gpu\PY\3
6\tensorflow\core\common_runtime\bfc_allocator.cc:680] Stats:
Limit: 1631711232
InUse: 1229069056
MaxInUse: 1631711232
NumAllocs: 51
MaxAllocSize: 422898176

2018-03-13 15:20:04.122000: W C:\tf_jenkins\workspace\rel-win\M\windows-gpu\PY\3
6\tensorflow\core\common_runtime\bfc_allocator.cc:279] _______________________
**************************************************************************x
2018-03-13 15:20:04.123000: W C:\tf_jenkins\workspace\rel-win\M\windows-gpu\PY\3
6\tensorflow\core\framework\op_kernel.cc:1202] OP_REQUIRES failed at assign_op.h
:111 : Resource exhausted: OOM when allocating tensor with shape[98304,1024] and
type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_b
fc
Traceback (most recent call last):
File "D:\Program Files (x86)\anaconda\lib\site-packages\tensorflow\python\clie
nt\session.py", line 1361, in _do_call
return fn(*args)
File "D:\Program Files (x86)\anaconda\lib\site-packages\tensorflow\python\clie
nt\session.py", line 1340, in _run_fn
target_list, status, run_metadata)
File "D:\Program Files (x86)\anaconda\lib\site-packages\tensorflow\python\fram
ework\errors_impl.py", line 516, in exit
c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocat
ing tensor with shape[98304,1024] and type float on /job:localhost/replica:0/tas
k:0/device:GPU:0 by allocator GPU_0_bfc
[[Node: save/Assign_7 = Assign[T=DT_FLOAT, _class=["loc:@Variable_2"],
use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/
device:GPU:0"](Variable_2/Adam, save/RestoreV2/_23)]]
Hint: If you want to see a list of allocated tensors when OOM happens, add repor
t_tensor_allocations_upon_oom to RunOptions for current allocation info.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "run_model.py", line 117, in
saver.restore(sess, "./model/model.ckpt")
File "D:\Program Files (x86)\anaconda\lib\site-packages\tensorflow\python\trai
ning\saver.py", line 1755, in restore
{self.saver_def.filename_tensor_name: save_path})
File "D:\Program Files (x86)\anaconda\lib\site-packages\tensorflow\python\clie
nt\session.py", line 905, in run
run_metadata_ptr)
File "D:\Program Files (x86)\anaconda\lib\site-packages\tensorflow\python\clie
nt\session.py", line 1137, in _run
feed_dict_tensor, options, run_metadata)
File "D:\Program Files (x86)\anaconda\lib\site-packages\tensorflow\python\clie
nt\session.py", line 1355, in _do_run
options, run_metadata)
File "D:\Program Files (x86)\anaconda\lib\site-packages\tensorflow\python\clie
nt\session.py", line 1374, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocat
ing tensor with shape[98304,1024] and type float on /job:localhost/replica:0/tas
k:0/device:GPU:0 by allocator GPU_0_bfc
[[Node: save/Assign_7 = Assign[T=DT_FLOAT, _class=["loc:@Variable_2"],
use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/
device:GPU:0"](Variable_2/Adam, save/RestoreV2/_23)]]
Hint: If you want to see a list of allocated tensors when OOM happens, add repor
t_tensor_allocations_upon_oom to RunOptions for current allocation info.

Caused by op 'save/Assign_7', defined at:
File "run_model.py", line 113, in
saver=tf.train.Saver()
File "D:\Program Files (x86)\anaconda\lib\site-packages\tensorflow\python\trai
ning\saver.py", line 1293, in init
self.build()
File "D:\Program Files (x86)\anaconda\lib\site-packages\tensorflow\python\trai
ning\saver.py", line 1302, in build
self._build(self._filename, build_save=True, build_restore=True)
File "D:\Program Files (x86)\anaconda\lib\site-packages\tensorflow\python\trai
ning\saver.py", line 1339, in _build
build_save=build_save, build_restore=build_restore)
File "D:\Program Files (x86)\anaconda\lib\site-packages\tensorflow\python\trai
ning\saver.py", line 796, in _build_internal
restore_sequentially, reshape)
File "D:\Program Files (x86)\anaconda\lib\site-packages\tensorflow\python\trai
ning\saver.py", line 471, in _AddRestoreOps
assign_ops.append(saveable.restore(saveable_tensors, shapes))
File "D:\Program Files (x86)\anaconda\lib\site-packages\tensorflow\python\trai
ning\saver.py", line 161, in restore
self.op.get_shape().is_fully_defined())
File "D:\Program Files (x86)\anaconda\lib\site-packages\tensorflow\python\ops
state_ops.py", line 280, in assign
validate_shape=validate_shape)
File "D:\Program Files (x86)\anaconda\lib\site-packages\tensorflow\python\ops
gen_state_ops.py", line 61, in assign
use_locking=use_locking, name=name)
File "D:\Program Files (x86)\anaconda\lib\site-packages\tensorflow\python\fram
ework\op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "D:\Program Files (x86)\anaconda\lib\site-packages\tensorflow\python\fram
ework\ops.py", line 3271, in create_op
op_def=op_def)
File "D:\Program Files (x86)\anaconda\lib\site-packages\tensorflow\python\fram
ework\ops.py", line 1650, in init
self._traceback = self._graph._extract_stack() # pylint: disable=protected-
access

ResourceExhaustedError (see above for traceback): OOM when allocating tensor wit
h shape[98304,1024] and type float on /job:localhost/replica:0/task:0/device:GPU
:0 by allocator GPU_0_bfc
[[Node: save/Assign_7 = Assign[T=DT_FLOAT, _class=["loc:@Variable_2"],
use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/
device:GPU:0"](Variable_2/Adam, save/RestoreV2/_23)]]
Hint: If you want to see a list of allocated tensors when OOM happens, add repor
t_tensor_allocations_upon_oom to RunOptions for current allocation info.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions