CycleGAN

Problem

absence of target domain pair as in img2img
capture, problem with current GAN; mode collapse

 

Cycle consistency

if we translate word FR->EN and then EN->FR should be the same

G: X->Y F: Y->X G&F should be inverse

 

Objective function

 

1-adversarial loss to march distribution of generated image to data distribution
2-cycle consistency loss to prevent mappings G&F from contradicting each other

 

Novelty

• Nearly same architecture as img2img
• in addition to Enforce cycle consistency

Image-to-Image Translation with Conditional Adversarial Networks

problem with old approaches is finding effective loss ( what to ) as CNN use Ecludian distance which can be easily minimised by avg all possible outputs resulting in blurry results.
coming up with a loss/objective function is hard.

 

Approach

• use GANs as objective can be to generate realistic undistinguishable results
• Gan learns “structured loss” which penalizes the joint configuration of the output.

 

Related

• cGANs (labels, text, images) for applications like painting, future frame prediction, style transfer

Architecture

• Generator uses U-net … Discriminator uses convolutional patchGAN classifier which penalizes structure @ scale of image patches.

 

Loss function

 

add l1 loss (less blurry) discriminator job remain same, while generator has one more

Tasks
1- fool discriminator
2-be near ground truth output

without Z .. net would map x=>y but deterministic ie: match no distinction, thus use Gaussian noise

Open-problem: cGAN that produce stochastic output to capture full entropy of the conditional distribution they model

Generator:
U-net with skip-connections allows input & output to share info directly eg:) image colorization tasks —input & output share information of prominent edges

 

Discriminator

L1&L2 produce blurry results on image generation, loss fails to encourage high-frequency crispness but capture the low-frequency accurately thus we only focus on high-frequency and use the l1 framework for low-frequency

PatchGAN: penalize structures at the scale of patches… classify if NXN patch is real or fake. runs as a convolutional across the image averaging the response to provide the ultimate output of D

more science : D models image as Markov-Random-Field\

cGan makes image more color, same as L1 ake it lurry when uncertain,

l1 when uncertain incentive average

 

Novelty

• cGAN
• U-net for generator
• PatchGAN for discriminator
• L1 loss in addition to the loss

Transferring microscopy modalities using cGAN

PC: show inner details,
DFC: show neighbouring cells
Both are complementary but needs 2 different scanners and time consuming

 

How to solve?….. GAN !

 

Network architecture

Generator:

• Unet- with skip connections
• noise added to decoder layers inform of dropout as it is ignored if added to the very beginning Discriminator:

• D1 & D2 use patchGAN to model high-level frequencies NxN patches runs conv and avg same as in pix2pix
• D1 for matching images
• D2 for matching mask with target image

 

Dataset & Training:

• 1600 pairs of 256×256 pixels
• 800 tuples (DIC,PC,Mask)
• patchgan: 70×70
• batchsize 1
• training iterations: 300

Evaluation: NRMSE & SSIM

Novelty:

• similar to pix2pix
• but with cell mask + extra Discriminator