Publications

2026

1. arXiv

   

   Tri-Prompting: Video Diffusion with Unified Control over Scene, Subject, and Motion

   Z. Zhou, X. Zhan, Z. Chen, S. Y. Kim, N. Zhao, H. Zheng, Q. Liu, H. Zhang, Z. Lin, Y. Zhou, and J. Luo

   arXiv preprint arXiv:2603.15614, 2026

   Abs arXiv Bib Website

   Recent video diffusion models have made remarkable strides in visual quality, yet precise, fine-grained control remains a key bottleneck that limits practical customizability for content creation. For AI video creators, three forms of control are crucial: (i) scene composition, (ii) multi-view consistent subject customization, and (iii) camera-pose or object-motion adjustment. Existing methods typically handle these dimensions in isolation, with limited support for multi-view subject synthesis and identity preservation under arbitrary pose changes. This lack of a unified architecture makes it difficult to support versatile, jointly controllable video. We introduce Tri-Prompting, a unified framework and two-stage training paradigm that integrates scene composition, multi-view subject consistency, and motion control. Our approach leverages a dual-condition motion module driven by 3D tracking points for background scenes and downsampled RGB cues for foreground subjects. To ensure a balance between controllability and visual realism, we further propose an inference ControlNet scale schedule. Tri-Prompting supports novel workflows, including 3D-aware subject insertion into any scenes and manipulation of existing subjects in an image. Experimental results demonstrate that Tri-Prompting significantly outperforms specialized baselines such as Phantom and DaS in multi-view subject identity, 3D consistency, and motion accuracy.

   @article{zhou2026triprompting,
     preview_zoom = {triprompting.gif},
     title = {Tri-Prompting: Video Diffusion with Unified Control over Scene, Subject, and Motion},
     author = {Zhou, Z. and Zhan, X. and Chen, Z. and Kim, S. Y. and Zhao, N. and Zheng, H. and Liu, Q. and Zhang, H. and Lin, Z. and Zhou, Y. and Luo, J.},
     journal = {arXiv preprint arXiv:2603.15614},
     year = {2026},
   }

2. CVPR

   

   HBridge: H-Shape Bridging of Heterogeneous Experts for Unified Multimodal Understanding and Generation

   X. Wang, Z. Zhang, H. Zhang, Z. Lin, Y. Zhou, Q. Liu, S. Zhang, Y. Li, S. Liu, H. Zheng, J. Kuen, Y. Wang, C. Gao, and N. Sang

   In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2026

   Abs arXiv Bib

   Recent unified models integrate understanding experts (e.g., LLMs) with generative experts (e.g., diffusion models), achieving strong multimodal performance. However, recent advanced methods such as BAGEL and LMFusion follow the Mixture-of-Transformers (MoT) paradigm, adopting a symmetric design that mirrors one expert to another for convenient initialization and fusion, which remains suboptimal due to inherent modality discrepancies. In this work, we propose HBridge, an asymmetric H-shaped architecture that enables heterogeneous experts to optimally leverage pretrained priors from their respective modality domains. Unlike prior dense fusion strategies that straightforwardly connect all layers between experts via shared attention, HBridge selectively bridges intermediate layers, reducing over 40% attention sharing, which improves efficiency and enhances generation quality. Shallow and deep layers, which capture modality-specific representations, are decoupled, while mid-layer bridging promotes semantic alignment. To further strengthen cross-modal coherence, we introduce semantic reconstruction tokens that explicitly guide the generative expert to reconstruct visual semantic tokens of the target image. Extensive experiments across multiple benchmarks demonstrate the effectiveness and superior performance of HBridge, establishing a new paradigm for unified multimodal generation.

   @inproceedings{wang2026hbridge,
     preview_zoom = {hbridge_zoom.jpg},
     title = {HBridge: H-Shape Bridging of Heterogeneous Experts for Unified Multimodal Understanding and Generation},
     author = {Wang, X. and Zhang, Z. and Zhang, H. and Lin, Z. and Zhou, Y. and Liu, Q. and Zhang, S. and Li, Y. and Liu, S. and Zheng, H. and Kuen, J. and Wang, Y. and Gao, C. and Sang, N.},
     booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
     year = {2026},
   }

2025

1. AAAI

   

   RealUHR: Harnessing Patch-Cascade Flows for Photorealistic Ultra-High-Resolution Synthesis

   Y. Yu, H. Zheng, Z. Lin, C. Barnes, Y. Zhou, and J. Luo

   In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), 2025

   Abs Bib HTML PDF

   Ultra-high-resolution (UHR) text-to-image synthesis faces significant hurdles, including immense computational costs and a scarcity of training data. To address these, we introduce RealUHR, an efficient and scalable framework for generating photorealistic 4K images. At its core, RealUHR employs a Patch-Cascade Flow Matching pipeline that ensures global coherence without costly patch fusion by initiating generation from a semantically meaningful structure. This enables highly efficient, few-step inference for independent patches. Our key contribution is Guidance-Consistent Adaptation (GCA), a novel two-stage strategy to resolve the fundamental objective mismatch in guidance-distilled models. GCA allows powerful backbones like FLUX to be effectively adapted for patch-aware UHR synthesis. The framework’s detail-rendering capabilities are further enhanced by a non-uniform time schedule. Experiments show that RealUHR establishes superior performance in both quality and efficiency, and excels in zero-shot applications such as creative up-sampling and generative artifact suppression.

   @inproceedings{yu2025realuhr,
     title = {RealUHR: Harnessing Patch-Cascade Flows for Photorealistic Ultra-High-Resolution Synthesis},
     author = {Yu, Y. and Zheng, H. and Lin, Z. and Barnes, C. and Zhou, Y. and Luo, J.},
     booktitle = {Proceedings of the AAAI Conference on Artificial Intelligence (AAAI)},
     year = {2025},
     preview_zoom = {realuhr_zoom.jpg}
   }

2. NeurIPS

   

   PixPerfect: Seamless Latent Diffusion Local Editing with Discriminative Pixel-Space Refinement

   H. Zheng, Y. Yao, Y. Yu, Y. Zhou, Z. Lin, and J. Luo

   In Advances in Neural Information Processing Systems (NeurIPS), 2025

   Abs arXiv Bib

   Latent Diffusion Models (LDMs) have markedly advanced the quality of image inpainting and local editing. However, the inherent latent compression often introduces pixel-level inconsistencies, such as chromatic shifts, texture mismatches, and visible seams along editing boundaries. Existing remedies, including background-conditioned latent decoding and pixel-space harmonization, usually fail to fully eliminate these artifacts in practice and do not generalize well across different latent representations or tasks. We introduce PixPerfect, a pixel-level refinement framework that delivers seamless, high-fidelity local edits across diverse LDM architectures and tasks. PixPerfect leverages a differentiable discriminative pixel space that amplifies and suppresses subtle color and texture discrepancies, a comprehensive artifact simulation pipeline that exposes the refiner to realistic local editing artifacts during training, and a direct pixel-space refinement scheme that ensures broad applicability across diverse latent representations and tasks. Extensive experiments on inpainting, object removal, and insertion benchmarks demonstrate that PixPerfect substantially enhances perceptual fidelity and downstream editing performance.

   @inproceedings{zheng2025pixperfect,
     preview_zoom = {pixperfect_zoom.jpg},
     title = {PixPerfect: Seamless Latent Diffusion Local Editing with Discriminative Pixel-Space Refinement},
     author = {Zheng, H. and Yao, Y. and Yu, Y. and Zhou, Y. and Lin, Z. and Luo, J.},
     booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
     year = {2025},
   }

3. ICCV

   

   OmniPaint: Mastering Object-Oriented Editing via Disentangled Insertion-Removal Inpainting

   Y. Yu, Z. Zeng, H. Zheng, and J. Luo

   In IEEE/CVF International Conference on Computer Vision (ICCV), 2025

   Abs arXiv Bib Website

   Diffusion-based generative models have revolutionized object-oriented image editing, yet their deployment in realistic object removal and insertion remains hampered by challenges such as the intricate interplay of physical effects and insufficient paired training data. In this work, we introduce OmniPaint, a unified framework that re-conceptualizes object removal and insertion as interdependent processes rather than isolated tasks. Leveraging a pre-trained diffusion prior along with a progressive training pipeline comprising initial paired sample optimization and subsequent large-scale unpaired refinement via CycleFlow, OmniPaint achieves precise foreground elimination and seamless object insertion while faithfully preserving scene geometry and intrinsic properties. Furthermore, our novel CFD metric offers a robust, reference-free evaluation of context consistency and object hallucination, establishing a new benchmark for high-fidelity image editing.

   @inproceedings{yu2025omnipaint,
     title = {OmniPaint: Mastering Object-Oriented Editing via Disentangled Insertion-Removal Inpainting},
     author = {Yu, Y. and Zeng, Z. and Zheng, H. and Luo, J.},
     booktitle = {IEEE/CVF International Conference on Computer Vision (ICCV)},
     year = {2025},
   }

4. ICCV

   

   DOLLAR: Few-Step Video Generation via Distillation and Latent Reward Optimization

   Z. Ding, C. Jin, D. Liu, H. Zheng, K. Singh, Q. Zhang, Y. Kang, Z. Lin, and Y. Liu

   In IEEE/CVF International Conference on Computer Vision (ICCV), 2025

   Abs arXiv Bib

   Diffusion probabilistic models have shown significant progress in video generation; however, their computational efficiency is limited by the large number of sampling steps required. Reducing sampling steps often compromises video quality or generation diversity. In this work, we introduce a distillation method that combines variational score distillation and consistency distillation to achieve few-step video generation, maintaining both high quality and diversity. We also propose a latent reward model fine-tuning approach to further enhance video generation performance according to any specified reward metric. This approach reduces memory usage and does not require the reward to be differentiable. Our method demonstrates state-of-the-art performance in few-step generation for 10-second videos (128 frames at 12 FPS). The distilled student model achieves a score of 82.57 on VBench, surpassing the teacher model as well as baseline models Gen-3, T2V-Turbo, and Kling. One-step distillation accelerates the teacher model’s diffusion sampling by up to 278.6 times, enabling near real-time generation.

   @inproceedings{ding2025dollar,
     title = {DOLLAR: Few-Step Video Generation via Distillation and Latent Reward Optimization},
     author = {Ding, Z. and Jin, C. and Liu, D. and Zheng, H. and Singh, K. and Zhang, Q. and Kang, Y. and Lin, Z. and Liu, Y.},
     booktitle = {IEEE/CVF International Conference on Computer Vision (ICCV)},
     year = {2025},
   }

5. CVPR

   

   TurboFill: Adapting Few-step Text-to-image Model for Fast Image Inpainting

   L. Xie, D. Pakhomov, Z. Wang, Z. Wu, Z. Chen, Y. Zhou, H. Zheng, Z. Zhang, Z. Lin, J. Zhou, and C. Dong

   In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025

   Abs arXiv Bib Website

   This paper introduces TurboFill, a fast image inpainting model that enhances a few-step text-to-image diffusion model with an inpainting adapter for high-quality and efficient inpainting. While standard diffusion models generate high-quality results, they incur high computational costs. We overcome this by training an inpainting adapter on a few-step distilled text-to-image model, DMD2, using a novel 3-step adversarial training scheme to ensure realistic, structurally consistent, and visually harmonious inpainted regions. To evaluate TurboFill, we propose two benchmarks: DilationBench, which tests performance across mask sizes, and HumanBench, based on human feedback for complex prompts. Experiments show that TurboFill outperforms both multi-step BrushNet and few-step inpainting methods, setting a new benchmark for high-performance inpainting tasks.

   @inproceedings{xie2025turbofill,
     title = {TurboFill: Adapting Few-step Text-to-image Model for Fast Image Inpainting},
     author = {Xie, L. and Pakhomov, D. and Wang, Z. and Wu, Z. and Chen, Z. and Zhou, Y. and Zheng, H. and Zhang, Z. and Lin, Z. and Zhou, J. and Dong, C.},
     booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
     pages = {7613--7622},
     year = {2025},
     preview_zoom = {turbofill_zoom.jpg}
   }

6. CVPR

   

   MetaShadow: Object-Centered Shadow Detection, Removal, and Synthesis

   T. Wang, J. Zhang, H. Zheng, Z. Ding, S. Cohen, Z. Lin, W. Xiong, C.-W. Fu, L. Figueroa, and S. Y. Kim

   In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025

   Abs arXiv Bib

   Shadows are often under-considered or even ignored in image editing applications, limiting the realism of the edited results. In this paper, we introduce MetaShadow, a three-in-one versatile framework that enables detection, removal, and controllable synthesis of shadows in natural images in an object-centered fashion. MetaShadow combines the strengths of two cooperative components: Shadow Analyzer, for object-centered shadow detection and removal, and Shadow Synthesizer, for reference-based controllable shadow synthesis. Notably, we optimize the learning of the intermediate features from Shadow Analyzer to guide Shadow Synthesizer to generate more realistic shadows that blend seamlessly with the scene. Extensive evaluations on multiple shadow benchmark datasets show significant improvements of MetaShadow over the existing state-of-the-art methods on object-centered shadow detection, removal, and synthesis.

   @inproceedings{wang2025metashadow,
     title = {MetaShadow: Object-Centered Shadow Detection, Removal, and Synthesis},
     author = {Wang, T. and Zhang, J. and Zheng, H. and Ding, Z. and Cohen, S. and Lin, Z. and Xiong, W. and Fu, C.-W. and Figueroa, L. and Kim, S. Y.},
     booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
     year = {2025},
     preview_zoom = {metashadow_zoom.jpg}
   }

7. arXiv

   

   ZipIR: Latent Pyramid Diffusion Transformer for High-Resolution Image Restoration

   Y. Yu, H. Zheng, Z. Zhang, J. Zhang, Y. Zhou, C. Barnes, Y. Liu, W. Xiong, Z. Lin, and J. Luo

   arXiv preprint arXiv:2504.08591, 2025

   Abs arXiv Bib

   Recent progress in generative models has significantly improved image restoration capabilities, particularly through powerful diffusion models that offer remarkable recovery of semantic details and local fidelity. However, deploying these models at ultra-high resolutions faces a critical trade-off between quality and efficiency due to the computational demands of long-range attention mechanisms. To address this, we introduce ZipIR, a novel framework that enhances efficiency, scalability, and long-range modeling for high-res image restoration. ZipIR employs a highly compressed latent representation that compresses image 32x, effectively reducing the number of spatial tokens, and enabling the use of high-capacity models like the Diffusion Transformer (DiT). Toward this goal, we propose a Latent Pyramid VAE (LP-VAE) design that structures the latent space into sub-bands to ease diffusion training. Trained on full images up to 2K resolution, ZipIR surpasses existing diffusion-based methods, offering unmatched speed and quality in restoring high-resolution images from severely degraded inputs.

   @article{yu2025zipir,
     preview_zoom = {zipir_zoom.jpg},
     title = {ZipIR: Latent Pyramid Diffusion Transformer for High-Resolution Image Restoration},
     author = {Yu, Y. and Zheng, H. and Zhang, Z. and Zhang, J. and Zhou, Y. and Barnes, C. and Liu, Y. and Xiong, W. and Lin, Z. and Luo, J.},
     journal = {arXiv preprint arXiv:2504.08591},
     year = {2025},
   }

2024

1. SIGGRAPH

   

   Object-level Scene Deocclusion

   Z. Liu, Q. Liu, C. Chang, J. Zhang, D. Pakhomov, H. Zheng, Z. Lin, D. Cohen-Or, and C. Fu

   In ACM SIGGRAPH 2024 Conference Papers, 2024

   Abs arXiv Bib Website

   Deoccluding the hidden portions of objects in a scene is a formidable task, particularly when addressing real-world scenes. In this paper, we present a new self-supervised PArallel visible-to-COmplete diffusion framework, named PACO, a foundation model for object-level scene deocclusion. Leveraging the rich prior of pre-trained models, we first design the parallel variational autoencoder, which produces a full-view feature map that simultaneously encodes multiple complete objects, and the visible-to-complete latent generator, which learns to implicitly predict the full-view feature map from partial-view feature map and text prompts extracted from the incomplete objects in the input image. To train PACO, we create a large-scale dataset with 500k samples to enable self-supervised learning, avoiding tedious annotations of the amodal masks and occluded regions. At inference, we devise a layer-wise deocclusion strategy to improve efficiency while maintaining the deocclusion quality. Extensive experiments on COCOA and various real-world scenes demonstrate the superior capability of PACO for scene deocclusion, surpassing the state of the arts by a large margin.

   @inproceedings{liu2024deocclusion,
     title = {Object-level Scene Deocclusion},
     author = {Liu, Z. and Liu, Q. and Chang, C. and Zhang, J. and Pakhomov, D. and Zheng, H. and Lin, Z. and Cohen-Or, D. and Fu, C.},
     booktitle = {ACM SIGGRAPH 2024 Conference Papers},
     year = {2024},
   }

2. TPAMI

   

   Structure-Guided Image Inpainting with Image-level and Object-level Semantic Discriminators

   H. Zheng, Z. Lin, J. Lu, S. Cohen, E. Shechtman, J. Zhang, N. Xu, S. Amirghodsi, and J. Luo

   IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2024

   Abs arXiv Bib

   Structure-guided image completion aims to inpaint a local region of an image according to an input guidance map from users. While such a task enables many practical applications for interactive editing, existing methods often struggle to hallucinate realistic object instances in complex natural scenes. Such a limitation is partially due to the lack of semantic-level constraints inside the hole region as well as the lack of a mechanism to enforce realistic object generation. In this work, we propose a learning paradigm that consists of semantic discriminators and object-level discriminators for improving the generation of complex semantics and objects. Specifically, the semantic discriminators leverage pretrained visual features to improve the realism of the generated visual concepts. Moreover, the object-level discriminators take aligned instances as inputs to enforce the realism of individual objects. Our proposed scheme significantly improves the generation quality and achieves state-of-the-art results on various tasks, including segmentation-guided completion, edge-guided manipulation and panoptically-guided manipulation on Places2 datasets.

   @article{zheng2024structure,
     preview_zoom = {structure_zoom.jpg},
     title = {Structure-Guided Image Inpainting with Image-level and Object-level Semantic Discriminators},
     author = {Zheng, H. and Lin, Z. and Lu, J. and Cohen, S. and Shechtman, E. and Zhang, J. and Xu, N. and Amirghodsi, S. and Luo, J.},
     journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI)},
     year = {2024},
   }

2023

1. ICIP

   

   Point Cloud Denoising via Momentum Ascent in Gradient Fields

   Y. Zhao, H. Zheng, Z. Wang, J. Luo, and E. Y. Lam

   In IEEE International Conference on Image Processing (ICIP), 2023

   Abs arXiv Bib

   To achieve point cloud denoising, traditional methods heavily rely on geometric priors, and most learning-based approaches suffer from outliers and loss of details. Recently, the gradient-based method was proposed to estimate the gradient fields from the noisy point clouds using neural networks, and refine the position of each point according to the estimated gradient. However, the predicted gradient could fluctuate, leading to perturbed and unstable solutions, as well as a long inference time. To address these issues, we develop the momentum gradient ascent method that leverages the information of previous iterations in determining the trajectories of the points, thus improving the stability of the solution and reducing the inference time. Experiments demonstrate that the proposed method outperforms state-of-the-art approaches with a variety of point clouds, noise types, and noise levels.

   @inproceedings{zhao2023pointcloud,
     preview_zoom = {pointcloud_zoom.jpg},
     title = {Point Cloud Denoising via Momentum Ascent in Gradient Fields},
     author = {Zhao, Y. and Zheng, H. and Wang, Z. and Luo, J. and Lam, E. Y.},
     booktitle = {IEEE International Conference on Image Processing (ICIP)},
     year = {2023},
   }

2. ICIP

   

   Improving Video Colorization by Test-Time Tuning

   Y. Zhao, H. Zheng, J. Luo, and E. Y. Lam

   In IEEE International Conference on Image Processing (ICIP), 2023

   Abs arXiv Bib Code

   With the advancements in deep learning, video colorization by propagating color information from a colorized reference frame to a monochrome video sequence has been well explored. However, the existing approaches often suffer from overfitting the training dataset and sequentially lead to suboptimal performance on colorizing testing samples. To address this issue, we propose an effective method, which aims to enhance video colorization through test-time tuning. By exploiting the reference to construct additional training samples during testing, our approach achieves a performance boost of 1 3 dB in PSNR on average compared to the baseline. Code is available at: https://github.com/IndigoPurple/T3

   @inproceedings{zhao2023videocolor,
     preview_zoom = {videocolor_zoom.jpg},
     title = {Improving Video Colorization by Test-Time Tuning},
     author = {Zhao, Y. and Zheng, H. and Luo, J. and Lam, E. Y.},
     booktitle = {IEEE International Conference on Image Processing (ICIP)},
     year = {2023},
   }

3. ACM MM

   

   Jurassic World Remake: Bringing Ancient Fossils Back to Life via Zero-Shot Long Image-to-Image Translation

   A. Martin, H. Zheng, J. An, and J. Luo

   In ACM International Conference on Multimedia (ACM MM), 2023

   Abs arXiv Bib

   With a strong understanding of the target domain from natural language, we produce promising results in translating across large domain gaps and bringing skeletons back to life. In this work, we use text-guided latent diffusion models for zero-shot image-to-image translation (I2I) across large domain gaps (longI2I), where large amounts of new visual features and new geometry need to be generated to enter the target domain. Being able to perform translations across large domain gaps has a wide variety of real-world applications in criminology, astrology, environmental conservation, and paleontology. In this work, we introduce a new task Skull2Animal for translating between skulls and living animals. On this task, we find that unguided Generative Adversarial Networks (GANs) are not capable of translating across large domain gaps. Instead of these traditional I2I methods, we explore the use of guided diffusion and image editing models and provide a new benchmark model, Revive-2I, capable of performing zero-shot I2I via text-prompting latent diffusion models. We find that guidance is necessary for longI2I because, to bridge the large domain gap, prior knowledge about the target domain is needed. In addition, we find that prompting provides the best and most scalable information about the target domain as classifier-guided diffusion models require retraining for specific use cases and lack stronger constraints on the target domain because of the wide variety of images they are trained on.

   @inproceedings{martin2023jurassic,
     preview_zoom = {jurassic_zoom.jpg},
     title = {Jurassic World Remake: Bringing Ancient Fossils Back to Life via Zero-Shot Long Image-to-Image Translation},
     author = {Martin, A. and Zheng, H. and An, J. and Luo, J.},
     booktitle = {ACM International Conference on Multimedia (ACM MM)},
     year = {2023},
   }

2022

1. ECCV

   

   Image Inpainting with Cascaded Modulation GAN and Object-Aware Training

   H. Zheng, Z. Lin, J. Lu, S. Cohen, E. Shechtman, C. Barnes, J. Zhang, N. Xu, S. Amirghodsi, and J. Luo

   In European Conference on Computer Vision (ECCV), 2022

   Abs arXiv Bib Code

   Recent image inpainting methods have made great progress but often struggle to generate plausible image structures when dealing with large holes in complex images. This is partially due to the lack of effective network structures that can capture both the long-range dependency and high-level semantics of an image. We propose cascaded modulation GAN (CM-GAN), a new network design consisting of an encoder with Fourier convolution blocks that extract multi-scale feature representations from the input image with holes and a dual-stream decoder with a novel cascaded global-spatial modulation block at each scale level. In each decoder block, global modulation is first applied to perform coarse and semantic-aware structure synthesis, followed by spatial modulation to further adjust the feature map in a spatially adaptive fashion. In addition, we design an object-aware training scheme to prevent the network from hallucinating new objects inside holes, fulfilling the needs of object removal tasks in real-world scenarios.

   @inproceedings{zheng2022cmgan,
     title = {Image Inpainting with Cascaded Modulation GAN and Object-Aware Training},
     author = {Zheng, H. and Lin, Z. and Lu, J. and Cohen, S. and Shechtman, E. and Barnes, C. and Zhang, J. and Xu, N. and Amirghodsi, S. and Luo, J.},
     booktitle = {European Conference on Computer Vision (ECCV)},
     pages = {277--296},
     year = {2022},
     preview_zoom = {cmgan_zoom.jpg}
   }

2. TPAMI

   

   Semantic Layout Manipulation with High-Resolution Sparse Attention

   H. Zheng, Z. Lin, J. Lu, S. Cohen, J. Zhang, N. Xu, and J. Luo

   IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2022

   Abs arXiv Bib

   We tackle the problem of semantic image layout manipulation, which aims to manipulate an input image by editing its semantic label map. A core problem of this task is how to transfer visual details from the input images to the new semantic layout while making the resulting image visually realistic. Recent work on learning cross-domain correspondence has shown promising results for global layout transfer with dense attention-based warping. However, this method tends to lose texture details due to the resolution limitation and the lack of smoothness constraint of correspondence. To adapt this paradigm for the layout manipulation task, we propose a high-resolution sparse attention module that effectively transfers visual details to new layouts at a resolution up to 512x512. To further improve visual quality, we introduce a novel generator architecture consisting of a semantic encoder and a two-stage decoder for coarse-to-fine synthesis.

   @article{zheng2022semantic,
     title = {Semantic Layout Manipulation with High-Resolution Sparse Attention},
     author = {Zheng, H. and Lin, Z. and Lu, J. and Cohen, S. and Zhang, J. and Xu, N. and Luo, J.},
     journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI)},
     year = {2022},
     preview_zoom = {semlayout_zoom.jpg}
   }

3. CVPR

   

   SpaceEdit: Learning a Unified Editing Space for Open-Domain Image Color Editing

   J. Shi, N. Xu, H. Zheng, A. Smith, J. Luo, and C. Xu

   In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022

   Abs arXiv Bib

   Recently, large pretrained models (e.g., BERT, StyleGAN, CLIP) have shown great knowledge transfer and generalization capability on various downstream tasks within their domains. Inspired by these efforts, in this paper we propose a unified model for open-domain image editing focusing on color and tone adjustment of open-domain images while keeping their original content and structure. Our model learns a unified editing space that is more semantic, intuitive, and easy to manipulate than the operation space (e.g., contrast, brightness, color curve) used in many existing photo editing softwares. Our model belongs to the image-to-image translation framework which consists of an image encoder and decoder, and is trained on pairs of before- and after-images to produce multimodal outputs.

   @inproceedings{shi2022spaceedit,
     title = {SpaceEdit: Learning a Unified Editing Space for Open-Domain Image Color Editing},
     author = {Shi, J. and Xu, N. and Zheng, H. and Smith, A. and Luo, J. and Xu, C.},
     booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
     pages = {19730--19739},
     year = {2022},
   }

4. ICIP

   

   MANet: Improving Video Denoising with a Multi-Alignment Network

   Y. Zhao, H. Zheng, Z. Wang, J. Luo, and E. Y. Lam

   In IEEE International Conference on Image Processing (ICIP), 2022

   Abs arXiv Bib Code

   In video denoising, the adjacent frames often provide very useful information, but accurate alignment is needed before such information can be harnessed. In this work, we present a multi-alignment network, which generates multiple flow proposals followed by attention-based averaging. It serves to mimic the non-local mechanism, suppressing noise by averaging multiple observations. Our approach can be applied to various state-of-the-art models that are based on flow estimation. Experiments on a large-scale video dataset demonstrate that our method improves the denoising baseline model by 0.2dB, and further reduces the parameters by 47% with model distillation.

   @inproceedings{zhao2022manet,
     title = {MANet: Improving Video Denoising with a Multi-Alignment Network},
     author = {Zhao, Y. and Zheng, H. and Wang, Z. and Luo, J. and Lam, E. Y.},
     booktitle = {IEEE International Conference on Image Processing (ICIP)},
     year = {2022},
   }

5. ICPR

   

   Learning to Aggregate and Refine Noisy Labels for Visual Sentiment Analysis

   W. Zhu, Z. Zheng, H. Zheng, H. Lyu, and J. Luo

   In International Conference on Pattern Recognition (ICPR), 2022

   Abs arXiv Bib

   Visual sentiment analysis has received increasing attention in recent years. However, the dataset’s quality is a concern because the sentiment labels are crowd-sourcing, subjective, and prone to mistakes, and poses a severe threat to the data-driven models, especially the deep neural networks. The deep models would generalize poorly on the testing cases when trained to over-fit the training samples with noisy sentiment labels. Inspired by the recent progress on learning with noisy labels, we propose a robust learning method to perform robust visual sentiment analysis. Our method relies on external memory to aggregate and filter noisy labels during training. The memory is composed of the prototypes with corresponding labels, which can be updated online.

   @inproceedings{zhu2022noisylabels,
     preview_zoom = {noisylabels_zoom.jpg},
     title = {Learning to Aggregate and Refine Noisy Labels for Visual Sentiment Analysis},
     author = {Zhu, W. and Zheng, Z. and Zheng, H. and Lyu, H. and Luo, J.},
     booktitle = {International Conference on Pattern Recognition (ICPR)},
     pages = {571--577},
     year = {2022},
   }

6. CICAI

   

   Cross-Camera Deep Colorization

   Y. Zhao, H. Zheng, M. Ji, and R. Huang

   In CAAI International Conference on Artificial Intelligence (CICAI), 2022

   Abs arXiv Bib

   In this paper, we consider the color-plus-mono dual-camera system and propose an end-to-end convolutional neural network to align and fuse images from it in an efficient and cost-effective way. Our method takes cross-domain and cross-scale images as input, and consequently synthesizes HR colorization results to facilitate the trade-off between spatial-temporal resolution and color depth in the single-camera imaging system. In contrast to the previous colorization methods, ours can adapt to color and monochrome cameras with distinctive spatial-temporal resolutions, rendering the flexibility and robustness in practical applications. The key ingredient of our method is a cross-camera alignment module that generates multi-scale correspondences for cross-domain image alignment. Through extensive experiments on various datasets and multiple settings, we validate the flexibility and effectiveness of our approach. Remarkably, our method consistently achieves substantial improvements, i.e., around 10dB PSNR gain, upon the state-of-the-art methods.

   @inproceedings{zhao2022crosscamera,
     preview_zoom = {crosscamera_zoom.jpg},
     title = {Cross-Camera Deep Colorization},
     author = {Zhao, Y. and Zheng, H. and Ji, M. and Huang, R.},
     booktitle = {CAAI International Conference on Artificial Intelligence (CICAI)},
     year = {2022},
   }

2021

1. Big Data

   

   Personalized Fashion Recommendation from Personal Social Media Data: An Item-to-Set Metric Learning Approach

   H. Zheng, K. Wu, J. Park, W. Zhu, and J. Luo

   In IEEE International Conference on Big Data (Big Data), 2021

   Abs arXiv Bib Code

   With the growth of online shopping for fashion products, accurate fashion recommendation has become a critical problem. Meanwhile, social networks provide an open and new data source for personalized fashion analysis. In this work, we study the problem of personalized fashion recommendation from social media data, i.e. recommending new outfits to social media users that fit their fashion preferences. To this end, we present an item-to-set metric learning framework that learns to compute the similarity between a set of historical fashion items of a user to a new fashion item. To extract features from multi-modal street-view fashion items, we propose an embedding module that performs multi-modality feature extraction and cross-modality gated fusion.

   @inproceedings{zheng2021fashion,
     title = {Personalized Fashion Recommendation from Personal Social Media Data: An Item-to-Set Metric Learning Approach},
     author = {Zheng, H. and Wu, K. and Park, J. and Zhu, W. and Luo, J.},
     booktitle = {IEEE International Conference on Big Data (Big Data)},
     pages = {5014--5023},
     year = {2021},
   }

2. ICCV

   

   Learning Bias-Invariant Representation by Cross-Sample Mutual Information Minimization

   W. Zhu, H. Zheng, H. Liao, W. Li, and J. Luo

   In IEEE/CVF International Conference on Computer Vision (ICCV), 2021

   Abs arXiv Bib

   Deep learning algorithms mine knowledge from the training data and thus would likely inherit the dataset’s bias information. As a result, the obtained model would generalize poorly and even mislead the decision process in real-life applications. We propose to remove the bias information misused by the target task with a cross-sample adversarial debiasing (CSAD) method. CSAD explicitly extracts target and bias features disentangled from the latent representation generated by a feature extractor and then learns to discover and remove the correlation between the target and bias features. The correlation measurement plays a critical role in adversarial debiasing and is conducted by a cross-sample neural mutual information estimator.

   @inproceedings{zhu2021biasinvariant,
     title = {Learning Bias-Invariant Representation by Cross-Sample Mutual Information Minimization},
     author = {Zhu, W. and Zheng, H. and Liao, H. and Li, W. and Luo, J.},
     booktitle = {IEEE/CVF International Conference on Computer Vision (ICCV)},
     pages = {15002--15012},
     year = {2021},
   }

2020

1. ECCV

   

   Example-Guided Image Synthesis using Masked Spatial-Channel Attention and Self-Supervision

   H. Zheng, H. Liao, L. Chen, W. Xiong, and J. Luo

   In European Conference on Computer Vision (ECCV), 2020

   Abs arXiv Bib Code

   Example-guided image synthesis has recently been attempted to synthesize an image from a semantic label map and an exemplary image. In the task, the additional exemplar image provides the style guidance that controls the appearance of the synthesized output. Despite the controllability advantage, the existing models are designed on datasets with specific and roughly aligned objects. In this paper, we tackle a more challenging and general task, where the exemplar is an arbitrary scene image that is semantically different from the given label map. To this end, we first propose a Masked Spatial-Channel Attention (MSCA) module which models the correspondence between two arbitrary scenes via efficient decoupled attention. Next, we propose an end-to-end network for joint global and local feature alignment and synthesis.

   @inproceedings{zheng2020exampleguided,
     preview_zoom = {exampleguided_zoom.jpg},
     title = {Example-Guided Image Synthesis using Masked Spatial-Channel Attention and Self-Supervision},
     author = {Zheng, H. and Liao, H. and Chen, L. and Xiong, W. and Luo, J.},
     booktitle = {European Conference on Computer Vision (ECCV)},
     pages = {422--439},
     year = {2020},
   }

2. TIP

   

   Pose Flow Learning from Person Images for Pose Guided Synthesis

   H. Zheng, L. Chen, C. Xu, and J. Luo

   IEEE Transactions on Image Processing (TIP), 2020

   Abs arXiv Bib Code

   Pose guided synthesis aims to generate a new image in an arbitrary target pose while preserving the appearance details from the source image. Existing approaches rely on either hard-coded spatial transformations or 3D body modeling. They often overlook complex non-rigid pose deformation or unmatched occluded regions, thus fail to effectively preserve appearance information. In this paper, we propose an unsupervised pose flow learning scheme that learns to transfer the appearance details from the source image. Based on such learned pose flow, we proposed GarmentNet and SynthesisNet, both of which use multi-scale feature-domain alignment for coarse-to-fine synthesis.

   @article{zheng2020poseflow,
     title = {Pose Flow Learning from Person Images for Pose Guided Synthesis},
     author = {Zheng, H. and Chen, L. and Xu, C. and Luo, J.},
     journal = {IEEE Transactions on Image Processing (TIP)},
     volume = {30},
     pages = {1898--1909},
     year = {2020},
   }

3. TPAMI

   

   CrossNet++: Cross-scale Large-parallax Warping for Reference-based Super-resolution

   Y. Tan, H. Zheng, Y. Zhu, and L. Fang

   IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2020

   Abs Bib HTML

   The ability of camera arrays to efficiently capture higher space-bandwidth product than single cameras has led to various multiscale and hybrid systems. These systems play vital roles in computational photography, including light field imaging, 360 VR camera, gigapixel videography, etc. One of the critical tasks in multiscale hybrid imaging is matching and fusing cross-resolution images from different cameras under perspective parallax. In this paper, we investigate the reference-based super-resolution (RefSR) problem associated with dual-camera or multi-camera systems. RefSR consists of super-resolving a low-resolution (LR) image given an external high-resolution (HR) reference image, where they suffer both a significant resolution gap (8×) and large parallax (∼10% pixel displacement). We present CrossNet++, an end-to-end network containing novel two-stage cross-scale warping modules, image encoder and fusion decoder. The stage I learns to narrow down the parallax distinctively with the strong guidance of landmarks and intensity distribution consensus. Then the stage II operates more fine-grained alignment and aggregation in feature domain to synthesize the final super-resolved image. To further address the large parallax, new hybrid loss functions comprising warping loss, landmark loss and super-resolution loss are proposed to regularize training and enable better convergence. CrossNet++ significantly outperforms the state-of-art on light field datasets as well as real dual-camera data. We further demonstrate the generalization of our framework by transferring it to video super-resolution and video denoising.

   @article{tan2020crossnetpp,
     preview_zoom = {crossnetpp_zoom.jpg},
     title = {CrossNet++: Cross-scale Large-parallax Warping for Reference-based Super-resolution},
     author = {Tan, Y. and Zheng, H. and Zhu, Y. and Fang, L.},
     journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI)},
     volume = {43},
     number = {12},
     pages = {4291--4305},
     year = {2020},
   }

4. ICPR

   Cost-Effective Adversarial Attacks against Scene Text Recognition

   M. Yang, H. Zheng, X. Bai, and J. Luo

   In International Conference on Pattern Recognition (ICPR), 2020

   Abs Bib HTML

   Scene text recognition is a challenging task due to the diversity in text appearance and complexity of natural scenes. Thanks to the development of deep learning and the large volume of training data, scene text recognition has made impressive progress in recent years. However, recent research on adversarial examples has shown that deep learning models are vulnerable to adversarial input with often imperceptible changes. As one of the most practical tasks in computer vision, scene text recognition is also facing huge security risks. To our best knowledge, there has been no work on adversarial attacks against scene text recognition. To investigate its effects on scene text recognition, we make the first attempt to attack the state-of-the-art scene text recognizers, i.e., attention-based recognizers. To that end, we first adjust the objective function designed for non-sequential tasks, such as image classification, semantic segmentation and image retrieval, to the sequential form. We then propose a novel and effective objective function to further reduce the amount of perturbation while achieving a higher attack success rate. Comprehensive experiments on several standard benchmarks clearly demonstrate effective adversarial effects on scene text recognition by the proposed attacks. It is noteworthy that while this is harmful for a recognition system, it is highly advantageous for a text-based CAPTCHA system.

   @inproceedings{yang2020adversarial,
     title = {Cost-Effective Adversarial Attacks against Scene Text Recognition},
     author = {Yang, M. and Zheng, H. and Bai, X. and Luo, J.},
     booktitle = {International Conference on Pattern Recognition (ICPR)},
     pages = {2368--2374},
     year = {2020},
   }

5. ACM MM

   

   Image Sentiment Transfer

   T. Chen, W. Xiong, H. Zheng, and J. Luo

   In ACM International Conference on Multimedia (ACM MM), 2020

   Abs arXiv Bib

   In this work, we introduce an important but still unexplored research task – image sentiment transfer. Compared with other related tasks that have been well-studied, such as image-to-image translation and image style transfer, transferring the sentiment of an image is more challenging. Given an input image, the rule to transfer the sentiment of each contained object can be completely different, making existing approaches that perform global image transfer by a single reference image inadequate to achieve satisfactory performance. In this paper, we propose an effective and flexible framework that performs image sentiment transfer at the object level. It first detects the objects and extracts their pixel-level masks, and then performs object-level sentiment transfer guided by multiple reference images for the corresponding objects.

   @inproceedings{chen2020sentiment,
     title = {Image Sentiment Transfer},
     author = {Chen, T. and Xiong, W. and Zheng, H. and Luo, J.},
     booktitle = {ACM International Conference on Multimedia (ACM MM)},
     pages = {4407--4415},
     year = {2020},
   }

2018

1. ECCV

   

   CrossNet: An End-to-end Reference-based Super Resolution Network using Cross-scale Warping

   H. Zheng, M. Ji, H. Wang, Y. Liu, and L. Fang

   In European Conference on Computer Vision (ECCV), 2018

   Abs arXiv Bib Code

   The Reference-based Super-resolution (RefSR) super-resolves a low-resolution (LR) image given an external high-resolution (HR) reference image, where the reference image and LR image share similar viewpoint but with significant resolution gap (8x). Existing RefSR methods work in a cascaded way such as patch matching followed by synthesis pipeline with two independently defined objective functions, leading to the inter-patch misalignment, grid effect and inefficient optimization. To resolve these issues, we present CrossNet, an end-to-end and fully-convolutional deep neural network using cross-scale warping. Our network contains image encoders, cross-scale warping layers, and fusion decoder, which improves the existing schemes both in precision (around 2dB-4dB) and efficiency (more than 100 times faster).

   @inproceedings{zheng2018crossnet,
     preview_zoom = {crossnet_zoom.jpg},
     title = {CrossNet: An End-to-end Reference-based Super Resolution Network using Cross-scale Warping},
     author = {Zheng, H. and Ji, M. and Wang, H. and Liu, Y. and Fang, L.},
     booktitle = {European Conference on Computer Vision (ECCV)},
     pages = {88--104},
     year = {2018},
   }

2017

1. BMVC

   

   Learning Cross-scale Correspondence and Patch-based Synthesis for Reference-based Super-Resolution

   H. Zheng, M. Ji, Z. Xu, H. Wang, Y. Liu, and L. Fang

   In British Machine Vision Conference (BMVC), 2017

   Abs Bib PDF

   In this paper, we explore the Reference-based Super-Resolution (RefSR) problem, which aims to super-resolve a low definition (LR) input to a high definition (HR) output, given another HR reference image that shares similar viewpoint or capture time with the LR input. We solve this problem by proposing a learning-based scheme, denoted as RefSR-Net. Specifically, we first design a Cross-scale Correspondence Network (CCNet) to indicate the cross-scale patch matching between reference and LR image. The CC-Net is formulated as a classification problem which predicts the correct matches from the candidate patches within the search range. Using dilated convolution, the training and feature map generation are efficiently implemented. Given the reference patch selected via CC-Net, we further propose a Super-resolution image Synthesis Network (SS-Net) for the synthesis of the HR output, by fusing the LR patch and the reference patch at multiple scales. Experiments on MPI Sintel Dataset and Light-Field (LF) video dataset demonstrate our learned correspondence features outperform existing features, and our proposed RefSR-Net substantially outperforms conventional single image SR and exemplar-based SR approaches.

   @inproceedings{zheng2017crossscale,
     preview_zoom = {crossscale_zoom.jpg},
     title = {Learning Cross-scale Correspondence and Patch-based Synthesis for Reference-based Super-Resolution},
     author = {Zheng, H. and Ji, M. and Xu, Z. and Wang, H. and Liu, Y. and Fang, L.},
     booktitle = {British Machine Vision Conference (BMVC)},
     pages = {138.1--138.13},
     year = {2017},
   }

2. ICCV

   

   SurfaceNet: an End-to-end 3D Neural Network for Multiview Stereopsis

   M. Ji, G. Juergen, H. Zheng, Y. Liu, and L. Fang

   In IEEE International Conference on Computer Vision (ICCV), 2017

   Abs arXiv Bib Code

   This paper proposes an end-to-end learning framework for multiview stereopsis. We term the network SurfaceNet. It takes a set of images and their corresponding camera parameters as input and directly infers the 3D model. The key advantage of the framework is that both photo-consistency as well as geometric relations of the surface structure can be directly learned for the purpose of multiview stereopsis in an end-to-end fashion. SurfaceNet is a fully 3D convolutional network which is achieved by encoding the camera parameters together with the images in a 3D voxel representation. We evaluate SurfaceNet on the large-scale DTU benchmark.

   @inproceedings{ji2017surfacenet,
     title = {SurfaceNet: an End-to-end 3D Neural Network for Multiview Stereopsis},
     author = {Ji, M. and Juergen, G. and Zheng, H. and Liu, Y. and Fang, L.},
     booktitle = {IEEE International Conference on Computer Vision (ICCV)},
     pages = {2307--2315},
     year = {2017},
   }

3. ICCVW

   

   Combining Exemplar-based Approach and Learning-based Approach for Light Field Super-resolution Using a Hybrid Imaging System

   H. Zheng, M. Guo, Y. Liu, and L. Fang

   In IEEE International Conference on Computer Vision Workshops (ICCVW), 2017

   Abs Bib PDF

   We propose a new method to super-resolve images captured by a hybrid light field system that consists of a standard light field camera and a high-resolution standard camera. The high-resolution image is taken as a reference to help with super-resolving the low-resolution light field images. Our method combines an exemplar-based algorithm with the state-of-the-art single image super-resolution approach and draws on the strengths of both. Both quantitative and qualitative experiments show that our proposed method substantially outperforms existing methods on standard light field datasets in the challenging large parallax setting.

   @inproceedings{zheng2017lightfield,
     preview_zoom = {lightfield_zoom.jpg},
     title = {Combining Exemplar-based Approach and Learning-based Approach for Light Field Super-resolution Using a Hybrid Imaging System},
     author = {Zheng, H. and Guo, M. and Liu, Y. and Fang, L.},
     booktitle = {IEEE International Conference on Computer Vision Workshops (ICCVW)},
     pages = {2481--2486},
     year = {2017},
   }

4. GlobalSIP

   

   Utilizing High-level Visual Feature for Indoor Shopping Mall Navigation

   Z. Xu, H. Zheng, M. Pang, Y. Zhu, X. Su, G. Zhou, and L. Fang

   In IEEE Global Conference on Signal and Information Processing (GlobalSIP), 2017

   Abs arXiv Bib

   Towards robust and convenient indoor shopping mall navigation, we propose a novel learning-based scheme to utilize the high-level visual information from the storefront images captured by personal devices of users. Specifically, we decompose the visual navigation problem into localization and map generation respectively. Given a storefront input image, a novel feature fusion scheme (denoted as FusionNet) is proposed by fusing the distinguishing DNN-based appearance feature and text feature for robust recognition of store brands, which serves for accurate localization. Regarding the map generation, we convert the user-captured indicator map of the shopping mall into a topological map by parsing the stores and their connectivity.

   @inproceedings{xu2017indoor,
     preview_zoom = {indoor_zoom.jpg},
     title = {Utilizing High-level Visual Feature for Indoor Shopping Mall Navigation},
     author = {Xu, Z. and Zheng, H. and Pang, M. and Zhu, Y. and Su, X. and Zhou, G. and Fang, L.},
     booktitle = {IEEE Global Conference on Signal and Information Processing (GlobalSIP)},
     pages = {1378--1382},
     year = {2017},
   }

2016

1. TMM

   

   Deep Learning for Surface Material Classification Using Haptic and Visual Information

   H. Zheng, L. Fang, M. Ji, M. Strese, Y. Oezer, and E. Steinbach

   IEEE Transactions on Multimedia (TMM), 2016

   Abs arXiv Bib Code

   When a user scratches a hand-held rigid tool across an object surface, an acceleration signal can be captured, which carries relevant information about the surface. More importantly, such a haptic signal is complementary to the visual appearance of the surface, which suggests the combination of both modalities for the recognition of the surface material. In this paper, we present a novel deep learning method dealing with the surface material classification problem based on a Fully Convolutional Network (FCN), which takes as input the aforementioned acceleration signal and a corresponding image of the surface texture. Experiments performed on the TUM surface material database demonstrate that our method achieves state-of-the-art classification accuracy robustly and efficiently.

   @article{zheng2016haptic,
     preview_zoom = {haptic_zoom.jpg},
     title = {Deep Learning for Surface Material Classification Using Haptic and Visual Information},
     author = {Zheng, H. and Fang, L. and Ji, M. and Strese, M. and Oezer, Y. and Steinbach, E.},
     journal = {IEEE Transactions on Multimedia (TMM)},
     volume = {18},
     pages = {2407--2416},
     year = {2016},
   }

2. TCSVT

   

   Computation and Memory Efficient Image Segmentation

   Y. Zhou, T. T. Do, H. Zheng, L. Fang, and N. M. Cheung

   IEEE Transactions on Circuits and Systems for Video Technology (TCSVT), 2016

   Abs Bib HTML

   In this paper, we address the segmentation problem under limited computation and memory resources. Given a segmentation algorithm, we propose a framework that can reduce its computation time and memory requirement simultaneously, while preserving its accuracy. The proposed framework uses standard pixel-domain downsampling and includes two main steps. Coarse segmentation is first performed on the downsampled image. Refinement is then applied to the coarse segmentation results. We make two novel contributions to enable competitive accuracy using this simple framework. First, we rigorously examine the effect of downsampling on segmentation using a signal processing analysis. The analysis helps to determine the uncertain regions, which are small image regions where pixel labels are uncertain after the coarse segmentation. Second, we propose an efficient minimum spanning tree-based algorithm to propagate the labels into the uncertain regions. We perform extensive experiments using several standard data sets. The experimental results show that our segmentation accuracy is comparable to state-of-the-art methods, while requiring much less computation time and memory than those methods.

   @article{zhou2016segmentation,
     preview_zoom = {segmentation_zoom.jpg},
     title = {Computation and Memory Efficient Image Segmentation},
     author = {Zhou, Y. and Do, T. T. and Zheng, H. and Fang, L. and Cheung, N. M.},
     journal = {IEEE Transactions on Circuits and Systems for Video Technology (TCSVT)},
     volume = {28},
     number = {1},
     pages = {46--61},
     year = {2016},
   }

2015

1. APSIPA

   

   Analysis of Sports Statistics via Graph-Signal Smoothness Prior

   H. Zheng, G. Cheung, and L. Fang

   In Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC), 2015

   Abs Bib PDF

   Since teams in a sporting league compete head-to-head according to a structured schedule, it is natural to interpret statistics emanating from competitions as signals on a graph modeling similarities among competing entities. In this paper, we analyse available sports statistics to predict game outcomes from a graph signal processing (GSP) perspective: GSP tools are used to remove (denoise) unwanted variability to reveal underlying predictable trends, and to interpolate missing data—predicted game outcomes in terms of point differential. First, we construct a graph for the desired graph-signal (point differential for every team pair): for an N-team league, we construct N subgraphs Gj, each containing N−1 nodes representing teams competing against opponent j. We next assign weight to each intra-subgraph edge based on similarity in observed statistics (e.g., total points scored, assists, etc) of the two connecting nodes (teams). We then connect nodes in different subgraphs representing the same teams, where the weight of an inter-subgraph edge connecting nodes in subgraphs Gk and Gl now reflects the similarity between opponents k and l. Finally, assuming a graph-signal smoothness prior, we compute the desired graph-signal on the constructed graph via an alternating convex programming procedure. Experimental results show that our graph-based scheme achieves better prediction than a competing k-nearest neighbor (kNN) scheme.

   @inproceedings{zheng2015sports,
     preview_zoom = {sports_zoom.jpg},
     title = {Analysis of Sports Statistics via Graph-Signal Smoothness Prior},
     author = {Zheng, H. and Cheung, G. and Fang, L.},
     booktitle = {Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC)},
     pages = {1071--1076},
     year = {2015},
   }

2. MLSP

   Preprocessing-free Surface Material Classification Using Convolutional Neural Networks Pretrained by Sparse Autoencoder

   M. Ji, L. Fang, H. Zheng, M. Strese, and E. Steinbach

   In IEEE International Workshop on Machine Learning for Signal Processing (MLSP), 2015

   Abs Bib HTML

   Acceleration signals captured during the interaction of a rigid tool with an object surface carry relevant information for surface material classification. Existing methods mostly rely on carefully designed perception-related features or features adapted from audio processing motivated by the observed similarity between acceleration signals and audio signals. In contrast, our proposed method automatically learns features from RAW acceleration data without preprocessing. The approach is based on Convolutional Neural Networks (CNN) trained and tested on RAW data. For better performance and faster convergence of the CNN, we use the weights of a trained sparse Autoencoder (AE) to initialize the weights of the first convolution layers of the CNN. This strategy is named CNN pretrained by sparse AE (ACNN). Our classification results on a publically available Haptic Texture Database demonstrate that the proposed algorithm performs favorably against existing methods.

   @inproceedings{ji2015acnn,
     title = {Preprocessing-free Surface Material Classification Using Convolutional Neural Networks Pretrained by Sparse Autoencoder},
     author = {Ji, M. and Fang, L. and Zheng, H. and Strese, M. and Steinbach, E.},
     booktitle = {IEEE International Workshop on Machine Learning for Signal Processing (MLSP)},
     pages = {1--6},
     year = {2015},
   }

2014

1. EMBC

   Early Melanoma Diagnosis with Mobile Imaging

   T. T. Do, Y. Zhou, H. Zheng, N. M. Cheung, and D. Koh

   In International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), 2014

   Abs Bib HTML

   We research a mobile imaging system for early diagnosis of melanoma. Different from previous work, we focus on smartphone-captured images, and propose a detection system that runs entirely on the smartphone. Smartphone-captured images taken under loosely-controlled conditions introduce new challenges for melanoma detection, while processing performed on the smartphone is subject to computation and memory constraints. To address these challenges, we propose to localize the skin lesion by combining fast skin detection and fusion of two fast segmentation results. We propose new features to capture color variation and border irregularity which are useful for smartphone-captured images. We also propose a new feature selection criterion to select a small set of good features used in the final lightweight system. Our evaluation confirms the effectiveness of proposed algorithms and features. In addition, we present our system prototype which computes selected visual features from a user-captured skin lesion image, and analyzes them to estimate the likelihood of malignance, all on an off-the-shelf smartphone.

   @inproceedings{do2014melanoma,
     title = {Early Melanoma Diagnosis with Mobile Imaging},
     author = {Do, T. T. and Zhou, Y. and Zheng, H. and Cheung, N. M. and Koh, D.},
     booktitle = {International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC)},
     pages = {6752--6757},
     year = {2014},
   }