python - Operations on huge dense matrices in numpy -


for purpose of training neural network, @ point have huge 212,243 × 2500 dense matrix phi, , vectors y (212243) , w (2500), stored numpy arrays of doubles. i'm trying compute is

w = dot(pinv(phi), y) # serialize w... r = dot(w, transpose(phi)) # serialize r...

my machine has 6 gb of ram , 16 gb of swap on ubuntu x64. started computation twice , twice has ended system (not python) swap errors after hour of work.

is there way perform computation on computer? doesn't need done python.

if don't need pseudoinverse else computing w, replace line with:

w = np.linalg.lstsq(phi, y)[0]

on system runs 2x faster, , uses half intermediate storage.


Comments

Post a Comment

Popular posts from this blog

linux - xterm copying to CLIPBOARD using copy-selection causes automatic updating of CLIPBOARD upon mouse selection -

i've tried various ways coerce xterm (versions 285 , 292) copy selection clipboard clipboard whenever press ctrl-shift-c. promising way far has been, in ~/.xresources, put this: xterm*vt100.translations: #override \ ctrl shift <keypress> c: copy-selection(clipboard) \n\ ctrl shift <keypress> v: insert-selection(clipboard) ctrl-shift-v works perfectly, copying has nuance... if restart xterm, highlighting text puts things in primary clipboard; expected, proper, default behavior. if hit ctrl-shift-c, copies current selection clipboard clipboard. the bug, however, if highlight text after first time pressed ctrl-shift-c, you'll see highlighting copies both primary and clipboard. cannot figure out how stop xterm updating clipboard upon selection... , doesn't make sense me in first place. i said, @ specific point in time, copy selection clipboard... yet starts updating automatically upon selection after doing once... anyone have workaround...
Read more

qt - Errors in generated MOC files for QT5 from cmake -

i generated moc files qt5 using set (cmake_automoc on) set(cmake_include_current_dir on) then add moc files src using set(src src/main.cpp src/video_widget_surface.cpp src/video_widget.cpp src/video_player.cpp #moc files moc/moc_video_player.cpp moc/moc_video_widget.cpp moc/moc_video_widget_surface.cpp finally add executable using add_executable(somegui ${src}) but errors in moc files saying : /other/workspace/perception/somestuff/moc/moc_video_widget.cpp:54:6: error: 'videowidget' has not been declared /other/workspace/perception/somestuff/moc/moc_video_widget.cpp:62:19: error: 'videowidget' has not been declared /other/workspace/perception/somestuff/moc/moc_video_widget.cpp:68:20: error: 'videowidget' has not been declared /other/workspace/perception/somestuff/moc/moc_video_widget.cpp:68:46: error: non-member function 'const qmetaobject* metaobject()' cannot have cv-qualifier /other/qt5.0.1/5.0.1/gcc_64/include/qtcore/qo...
Read more